* Unified project interface @ 2015-06-04 11:43 Dmitry Gutov 2015-06-04 14:40 ` Stephen Leake ` (2 more replies) 0 siblings, 3 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-06-04 11:43 UTC (permalink / raw) To: emacs-devel; +Cc: Jorgen Schaefer, Bozhidar Batsov Hi all, A while ago, there was floated a "unified project root" proposal. I'd like to resume that discussion. Unfortunately, the code proposed back them doesn't serve the immediate need I have in mind: basically, I need that function to return a *list* of directories, because both etags and elisp xref backends operate on multiple directories, via tags-table-list and load-path respectively. If we have it, we'll be able to untie `xref-find-regexp' from xref backends, because really there's nothing language-specific to this command. Now, before adding this feature to the core, are there other related general questions we want to be able to answer? Depending on that, a "project" might be better implemented as a cl-struct, providing specialized implementations for those several generic methods. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-04 11:43 Unified project interface Dmitry Gutov @ 2015-06-04 14:40 ` Stephen Leake 2015-06-05 0:08 ` Dmitry Gutov 2015-06-06 10:20 ` Bozhidar Batsov 2015-06-06 12:32 ` Eric Ludlam 2 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-06-04 14:40 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > A while ago, there was floated a "unified project root" proposal. I'd > like to resume that discussion. > > Unfortunately, the code proposed back them doesn't serve the immediate > need I have in mind: basically, I need that function to return a > *list* of directories, because both etags and elisp xref backends > operate on multiple directories, via tags-table-list and load-path > respectively. > > If we have it, we'll be able to untie `xref-find-regexp' from xref > backends, because really there's nothing language-specific to this > command. The list of directories that xref-find-regexp needs to search is not the project root; it is the list of directories included by the project source code. For that you need a function like "project-source-directories", which could be customized for each project backend. I handle this for ada-mode by defining a project file syntax. A project file defines a list of source directories, among other things. The list of source directories then gets stored in compilation-search-path. For ada-mode, project-source-directories would just return compilation-search-path. The ada-mode project file can be anywhere; in my projects, it is usually _not_ at the "project root directory", but down one or two layers in build/ or build/release/. I could add "prj_root" to the project file syntax; then ada-mode could provide a "unified project root" backend that knows what the current project file is, and returns the value from prj_root. But that would not be useful in ada-mode; that's why it's not there yet. The notion of "project root directory" is more useful for configuration management; typically a config management "project" consists of the tree of files under one root. But ada-mode knowns nothing about config management; that's vc's job. Hmm - vc could query for the current project root, and ada-mode would provide the answer. That might be useful, and a good reason to have a unified project interface. > Now, before adding this feature to the core, are there other related > general questions we want to be able to answer? Depending on that, a > "project" might be better implemented as a cl-struct, providing > specialized implementations for those several generic methods. There are lots of project meta-data that the ada-mode project file syntax provides; compiler options, object directory, case exceptions, etc. Some of those might be common with other projects. Getting a basic infrastructure in place first is a good idea; we can always promote things that turn out to be common among many backends. In any case, I believe a cl-struct is the way to go. I don't think we should try to unify the xref and project backends, but every major mode will probably want to provide both. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-04 14:40 ` Stephen Leake @ 2015-06-05 0:08 ` Dmitry Gutov 2015-06-05 10:08 ` Stephen Leake 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-06-05 0:08 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 06/04/2015 05:40 PM, Stephen Leake wrote: > The list of directories that xref-find-regexp needs to search is not > the project root; it is the list of directories included by the project > source code. Not exactly. xref-find-regexp is interested in all kinds of files, not just source files. That includes, say, the README file at the top of the project. > For that you need a function like "project-source-directories", which > could be customized for each project backend. However, you raise an interesting point. Whereas xref-find-references would like to search in all load-path directories, maybe it would be enough if xref-find-regexp only searches inside the current "project root", for some definition of that notion. I wonder how we could make it work this way. Make "project root" an orthogonal feature? > The ada-mode project file can be anywhere; in my projects, it is usually > _not_ at the "project root directory", but down one or two layers in > build/ or build/release/. We can't rely on every Elisp project declaring a "project root" in the same way, though. > Hmm - vc could query for the current project root, and ada-mode would > provide the answer. That might be useful, and a good reason to have a > unified project interface. VC is one option for this. Or the project may be not registered in any VCS yet, and the root would be determined based on, say, the presence of configure.ac. Again, this suggests that notions of "source directories" and "project root" can be somewhat orthogonal. On the other hand, after the project root is determined, it might want to add some new element(s) to the source directories list. > There are lots of project meta-data that the ada-mode project file syntax > provides; compiler options, object directory, case exceptions, etc. Some > of those might be common with other projects. Right. We'll need those pieces of metadata that are useful to more than one subsystem, and can be employed in a general way. Though a generic metadata storage might be useful as well: thus, if some minor mode has read the project file and parsed the compiler options, it can set the related metadata, so that code completion and linter could use it without waiting for Emacs core to standardize it. > I don't think we should try to unify the xref and project backends, but > every major mode will probably want to provide both. Mostly those major modes that can launch and interact with a smart external process, I think. The rest (except for Elisp) will leave it to minor modes. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-05 0:08 ` Dmitry Gutov @ 2015-06-05 10:08 ` Stephen Leake 2015-06-05 13:03 ` Stephen Leake ` (2 more replies) 0 siblings, 3 replies; 87+ messages in thread From: Stephen Leake @ 2015-06-05 10:08 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 06/04/2015 05:40 PM, Stephen Leake wrote: > >> The list of directories that xref-find-regexp needs to search is not >> the project root; it is the list of directories included by the project >> source code. > > Not exactly. xref-find-regexp is interested in all kinds of files, not > just source files. That includes, say, the README file at the top of > the project. No problem; make sure that directory is in project-source-directories. README is a perfectly valid source file; it is written in the "plain text" language (for which no compiler is needed ;). I have no problem with projects including more than one language in the sources; most of mine have Ada, Texinfo, LaTeX, text, elisp, and make source code. Which is one reason why project and xref cannot be merged (xref must be language-specific, even for tags). It might make sense to parameterize project-source-directories with a language (or major-mode) name, to get the corresponding subset. >> For that you need a function like "project-source-directories", which >> could be customized for each project backend. > > However, you raise an interesting point. Whereas xref-find-references > would like to search in all load-path directories, 'load-path' is an elisp notion. For other languages "source-path" is more appropriate. > maybe it would be enough if xref-find-regexp only searches inside the > current "project root", for some definition of that notion. No, xref-find-regexp should search project-source-directories. Most real projects include other projects, so limiting the search to only the top project is wrong in general (although that might be a useful option in some use cases). In addition, there might be a directory under project root that should _not_ be searched; the object file directory for ada-mode, for example. project-source-directories should return the union of the source directories for all of the included projects. That's what load-path is for elisp, and what compilation-search-path is for ada-mode and other language modes. For elisp, project-source-directories should simply return load-path. > I wonder how we could make it work this way. Make "project root" an > orthogonal feature? If you mean "orthogonal to source directories", then yes, that is what I am suggesting. >> The ada-mode project file can be anywhere; in my projects, it is usually >> _not_ at the "project root directory", but down one or two layers in >> build/ or build/release/. > > We can't rely on every Elisp project declaring a "project root" in the > same way, though. We can make it a requirement in order to use the general tool. But first we have to justify it; ada-mode has never needed that notion; neither has elisp. In my experience, only config management needs it. >> Hmm - vc could query for the current project root, and ada-mode would >> provide the answer. That might be useful, and a good reason to have a >> unified project interface. > > VC is one option for this. Or the project may be not registered in any > VCS yet, and the root would be determined based on, say, the presence > of configure.ac. Again, this suggests that notions of "source > directories" and "project root" can be somewhat orthogonal. Not just "somewhat"; "completely" :). It also points out that "project root" is poorly defined; I think that is because it's not at all clear when and why we need it. > On the other hand, after the project root is determined, it might want > to add some new element(s) to the source directories list. "it" is what here ? Can you give an example? >> There are lots of project meta-data that the ada-mode project file syntax >> provides; compiler options, object directory, case exceptions, etc. Some >> of those might be common with other projects. > > Right. We'll need those pieces of metadata that are useful to more > than one subsystem, and can be employed in a general way. Though a > generic metadata storage might be useful as well: thus, if some minor > mode has read the project file and parsed the compiler options, it can > set the related metadata, so that code completion and linter could use > it without waiting for Emacs core to standardize it. ada-mode uses a plist to represent the project metadata, and has examples of minor modes adding to the plist. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-05 10:08 ` Stephen Leake @ 2015-06-05 13:03 ` Stephen Leake 2015-06-05 13:14 ` Dmitry Gutov 2015-06-07 23:22 ` Dmitry Gutov 2015-06-07 23:15 ` Dmitry Gutov 2015-07-24 23:43 ` Dmitry Gutov 2 siblings, 2 replies; 87+ messages in thread From: Stephen Leake @ 2015-06-05 13:03 UTC (permalink / raw) To: emacs-devel Stephen Leake <stephen_leake@stephe-leake.org> writes: > Dmitry Gutov <dgutov@yandex.ru> writes: > >> On 06/04/2015 05:40 PM, Stephen Leake wrote: >> >>> The list of directories that xref-find-regexp needs to search is not >>> the project root; it is the list of directories included by the project >>> source code. >> >> Not exactly. xref-find-regexp is interested in all kinds of files, not >> just source files. That includes, say, the README file at the top of >> the project. > > No problem; make sure that directory is in project-source-directories. > > README is a perfectly valid source file; it is written in the "plain > text" language (for which no compiler is needed ;). > > I have no problem with projects including more than one language in the > sources; most of mine have Ada, Texinfo, LaTeX, text, elisp, and make > source code. Which is one reason why project and xref cannot be merged > (xref must be language-specific, even for tags). > > It might make sense to parameterize project-source-directories > with a language (or major-mode) name, to get the corresponding subset. There are a couple of use cases here that require different functions: 1) You are editing a LaTeX file containing documentation for a Python module implemented in Python and C++. You want to find all mentions/references related to feature "foo", in LaTeX, Python, and C++ source files. So you invoke project-find-regexp, which searches for *.tex, *.py, *.hhp, *.cpp in project-source-directories. 2) You are refactoring a C++ function foo::bar, and want to find all references to bar* for comparison. You invoke xref-find-regexp, which searches xref-source-directories. There are a couple of implementation options here: - implement project-source-directories as the union of xref-source-directories for all backends listed in a project file - implement xref-source-directories as a subset of project-source-directories for each backend. I'm inclined to the latter; it's closer to what ada-mode does now. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-05 13:03 ` Stephen Leake @ 2015-06-05 13:14 ` Dmitry Gutov 2015-06-08 1:24 ` Stephen Leake 2015-06-07 23:22 ` Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-06-05 13:14 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 06/05/2015 04:03 PM, Stephen Leake wrote: > 2) You are refactoring a C++ function foo::bar, and want to find all > references to bar* for comparison. You invoke xref-find-regexp, which > searches xref-source-directories. You'll call xref-find-references for that. Its default implementation will use a similar set of directories to xref-find-regexp, but it can be limited to the proper "source directories". Sorry, I'll comment on the rest later. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-05 13:14 ` Dmitry Gutov @ 2015-06-08 1:24 ` Stephen Leake 2015-06-09 18:16 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-06-08 1:24 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 06/05/2015 04:03 PM, Stephen Leake wrote: > >> 2) You are refactoring a C++ function foo::bar, and want to find all >> references to bar* for comparison. You invoke xref-find-regexp, which >> searches xref-source-directories. > > You'll call xref-find-references That handles a specific identifier, not all identifiers macthing "bar.*" (to use proper regexp syntax, rather than shell glob). > for that. Its default implementation > will use a similar set of directories to xref-find-regexp, but it can > be limited to the proper "source directories". How will you specify two (or more?) sets of directories for xref? -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-08 1:24 ` Stephen Leake @ 2015-06-09 18:16 ` Dmitry Gutov 2015-06-09 18:21 ` Eli Zaretskii 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-06-09 18:16 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 06/08/2015 04:24 AM, Stephen Leake wrote: > That handles a specific identifier, not all identifiers macthing "bar.*" > (to use proper regexp syntax, rather than shell glob). I suppose that kind of command could be implemented on top of the current xref API: first, find all identifiers that match this regexp (the "apropos" action, or maybe just use the identifier-completion-table), then find all references to each of them. Or probably have a new generic method, do the above in the default implementation and allow backends to override it. > How will you specify two (or more?) sets of directories for xref? Considering we don't currently specify even *one* set of directories xref can use (that is one of the goals of this discussion)... How about the same way as I'll specify that one, only twice? Two variables, two generic methods, or so on. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-09 18:16 ` Dmitry Gutov @ 2015-06-09 18:21 ` Eli Zaretskii 2015-06-09 18:49 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Eli Zaretskii @ 2015-06-09 18:21 UTC (permalink / raw) To: Dmitry Gutov; +Cc: stephen_leake, emacs-devel > From: Dmitry Gutov <dgutov@yandex.ru> > Date: Tue, 9 Jun 2015 21:16:35 +0300 > > On 06/08/2015 04:24 AM, Stephen Leake wrote: > > > That handles a specific identifier, not all identifiers macthing "bar.*" > > (to use proper regexp syntax, rather than shell glob). > > I suppose that kind of command could be implemented on top of the > current xref API: first, find all identifiers that match this regexp > (the "apropos" action, or maybe just use the > identifier-completion-table), then find all references to each of them. > > Or probably have a new generic method, do the above in the default > implementation and allow backends to override it. Both etags and id-utils support regexp search, so a separate method sounds like a better idea, IMO. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-09 18:21 ` Eli Zaretskii @ 2015-06-09 18:49 ` Dmitry Gutov 2015-06-09 19:03 ` Eli Zaretskii 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-06-09 18:49 UTC (permalink / raw) To: Eli Zaretskii; +Cc: stephen_leake, emacs-devel On 06/09/2015 09:21 PM, Eli Zaretskii wrote: > Both etags and id-utils support regexp search, so a separate method > sounds like a better idea, IMO. etags doesn't support search for references, which is what this subthread is about. id-utils does, though. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-09 18:49 ` Dmitry Gutov @ 2015-06-09 19:03 ` Eli Zaretskii 0 siblings, 0 replies; 87+ messages in thread From: Eli Zaretskii @ 2015-06-09 19:03 UTC (permalink / raw) To: Dmitry Gutov; +Cc: stephen_leake, emacs-devel > From: Dmitry Gutov <dgutov@yandex.ru> > Date: Tue, 9 Jun 2015 21:49:52 +0300 > Cc: stephen_leake@stephe-leake.org, emacs-devel@gnu.org > > On 06/09/2015 09:21 PM, Eli Zaretskii wrote: > > > Both etags and id-utils support regexp search, so a separate method > > sounds like a better idea, IMO. > > etags doesn't support search for references, which is what this > subthread is about. Depends on the program that created the database, I guess. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-05 13:03 ` Stephen Leake 2015-06-05 13:14 ` Dmitry Gutov @ 2015-06-07 23:22 ` Dmitry Gutov 2015-06-08 1:35 ` [SPAM UNSURE] " Stephen Leake 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-06-07 23:22 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 06/05/2015 04:03 PM, Stephen Leake wrote: > 1) You are editing a LaTeX file containing documentation for a Python > module implemented in Python and C++. You want to find all mentions/references > related to feature "foo", in LaTeX, Python, and C++ source files. So you > invoke project-find-regexp, which searches for *.tex, *.py, *.hhp, *.cpp > in project-source-directories. project-find-regexp? Do you imagine we'll need a separate command like that? > 2) You are refactoring a C++ function foo::bar, and want to find all > references to bar* for comparison. You invoke xref-find-regexp, which > searches xref-source-directories. Like mentioned, it'll hopefully be xref-find-references. > There are a couple of implementation options here: > > - implement project-source-directories as the union of > xref-source-directories for all backends listed in a project file This sounds like you envision a one-backend-per-language kind of situation. I think that there should be just one backend there, set in a minor mode, that knows enough about this mixed project's structure, to know where to search. > - implement xref-source-directories as a subset of > project-source-directories for each backend. > > I'm inclined to the latter; it's closer to what ada-mode does now. Could you explain what that means exactly? How is it implemented? ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [SPAM UNSURE] Re: Unified project interface 2015-06-07 23:22 ` Dmitry Gutov @ 2015-06-08 1:35 ` Stephen Leake 2015-06-09 19:04 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-06-08 1:35 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 06/05/2015 04:03 PM, Stephen Leake wrote: > >> 1) You are editing a LaTeX file containing documentation for a Python >> module implemented in Python and C++. You want to find all mentions/references >> related to feature "foo", in LaTeX, Python, and C++ source files. So you >> invoke project-find-regexp, which searches for *.tex, *.py, *.hhp, *.cpp >> in project-source-directories. > > project-find-regexp? Do you imagine we'll need a separate command like > that? Yes, that's the point. That's how you distinuish between multi-language and single-language searches. >> There are a couple of implementation options here: >> >> - implement project-source-directories as the union of >> xref-source-directories for all backends listed in a project file > > This sounds like you envision a one-backend-per-language kind of > situation. Yes, for xref. > I think that there should be just one backend there, set in > a minor mode, that knows enough about this mixed project's structure, > to know where to search. I don't see how that would be better; that minor mode would end up doing what I'm suggesting the "project" code do. >> - implement xref-source-directories as a subset of >> project-source-directories for each backend. >> >> I'm inclined to the latter; it's closer to what ada-mode does now. > > Could you explain what that means exactly? How is it implemented? I don't have a working example of this precise structure now. What I'm imagining is a "project file" that lists source directories without regard to language. To be concrete, imagine a project with the following structure: /root src *.c files doc *.tex files build foo.prj -- the project file *.make files foo.prj contains (among other things): src_dir=../src src_dir=../doc src_dir=. So project-source-directories is the list ("root/src" "root/doc" "root/build"). If we are in a .c buffer, and we request xref-source-directories, the code looks thru project-source-directories for directories that contain c files, and returns only ("root/src"). This could be an expensive check, so we'd want to cache the results. Alternately, foo.prj could have: language=c language=tex language=make src_dir(c)=../src src_dir(tex)=../doc src_dir(make)=. (You could get fancier and use "latex" as a language name, and have another statement that maps that to ".tex". Which is needed for C and Ada anyway, since they use two file extensions; .h/.c, .ads/.adb). That imposes the requirement on the user that they know what files are where. Note that the directories do not have to be language exclusive; they are here just for clarity. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [SPAM UNSURE] Re: Unified project interface 2015-06-08 1:35 ` [SPAM UNSURE] " Stephen Leake @ 2015-06-09 19:04 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-06-09 19:04 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 06/08/2015 04:35 AM, Stephen Leake wrote: > Yes, that's the point. That's how you distinuish between multi-language > and single-language searches. That is inadequate. An ideal "project" implementation should be smart enough to figure out whether to look for the symbol's definition in the files with the same extension as the current one, or in files with some other extensions as well. Elisp is lucky to have all its environment at hand. In many cases, for decent "find definitions" result, we'll need to know the project structure first anyway. >> I think that there should be just one backend there, set in >> a minor mode, that knows enough about this mixed project's structure, >> to know where to search. > > I don't see how that would be better; that minor mode would end up doing > what I'm suggesting the "project" code do. Maybe. Why wouldn't that be better? > I don't have a working example of this precise structure now. What I'm > imagining is a "project file" that lists source directories without > regard to language. To be concrete, imagine a project with the following > structure: That looks decent, but if you mean that xref-source-directories would take the whole set of source directories defined for that project and only take those from it that correspond to the current language, that seems unnecessarily limiting to me (unless there's a way it can be very smart about that, and the notion of "current language" will be generalized to possibly mean an arbitrary set of major modes and file extensions). Further, I don't want to have xref-source-directories as a separate variable or a function, unless we have to. The idea is to add a new feature to Emacs that deals with projects, and make xref use *that*. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-05 10:08 ` Stephen Leake 2015-06-05 13:03 ` Stephen Leake @ 2015-06-07 23:15 ` Dmitry Gutov 2015-06-08 1:59 ` Stephen Leake 2015-07-24 23:43 ` Dmitry Gutov 2 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-06-07 23:15 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 06/05/2015 01:08 PM, Stephen Leake wrote: > No problem; make sure that directory is in project-source-directories. How would that work, in technical terms? Which entity will "make sure" of that? Will a project-backend "source directories" implementation always cl-call-next-method? Will the default impl simply return the value of some variable, assigned by a major mode? > README is a perfectly valid source file; it is written in the "plain > text" language (for which no compiler is needed ;). I see a few problems with that: - README has no source code (it doesn't affect the logic of any program). "sources" already has a common meaning, and at least in one proprietary editor it's just one a subset of all project files (see the nomenclature here: https://www.jetbrains.com/idea/help/content-root.html). Further, if we can separate them, we could make sure xref-find-references only searches in the "proper" source directories. - There's no direct way to find the directory holding the README when you only have the value of load-path or compilation-search-path (because it's quite often one level above one of the path elements). So it would make sense to have a different name (and a different function/slot/etc) for directories that might be reached that way. > I have no problem with projects including more than one language in the > sources; most of mine have Ada, Texinfo, LaTeX, text, elisp, and make > source code. Which is one reason why project and xref cannot be merged > (xref must be language-specific, even for tags). xref should take the language into account, but it should work across them as well. For instance, a Ruby method call in an *.html.erb template file should get recognized as a reference to that method, and its "find definition" should lead to the corresponding *.rb file. > It might make sense to parameterize project-source-directories > with a language (or major-mode) name, to get the corresponding subset. That's probably not necessary: after all, only the default xref-find-references implementation will use project-source-directories in a naive way. A smarter xref backend should determine itself which directories to search. > 'load-path' is an elisp notion. For other languages "source-path" is > more appropriate. I didn't mean to suggest to call anything generic a "load path". > No, xref-find-regexp should search project-source-directories. Maybe even project-directories (project-roots?), if we agree to separate those notions. > Most real projects include other projects, so limiting the search to > only the top project is wrong in general (although that might be a > useful option in some use cases). Maybe when called with C-u. It's not easy to allow to specify multiple directories that way anyway. > In addition, there might be a directory under project root that should > _not_ be searched; the object file directory for ada-mode, for example. project-ignore-directories should be a thing. Probably in the format of grep-find-ignored-directories (and we might have a function corresponding to grep-find-ignored-files). Trouble is, projectile-ignored-directories seems to be more flexible already, so maybe that will have to be accompanied with an upgrade to how grep-find-ignored-directories are handled. > We can make it a requirement in order to use the general tool. But first > we have to justify it; ada-mode has never needed that notion; neither > has elisp. In my experience, only config management needs it. Like mentioned above, for example, the distinction would help with the default xref-find-references implementation. >> On the other hand, after the project root is determined, it might want >> to add some new element(s) to the source directories list. > > "it" is what here ? Can you give an example? Some package that defines that root-finding logic. Not every major mode knows where the relevant source directories are, or it might not know some of them. If we've opened one of the files in a third-party Elisp project that hasn't been loaded yet (so not in load-path), a project implementation can still add an element or several to project-source-directories, so that xref-find-references returns somewhat useful results anyway. Or, say, for Ruby: while the major mode might conceivably be taught about paths where to find system-wide libraries, the locations of source files in a given project depend on the project's type, and that must be something a project implementation knows better. Even if it'll simply add the root to project-source-directories. > ada-mode uses a plist to represent the project metadata, and has examples > of minor modes adding to the plist. Cool. Then it must be useful indeed. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-07 23:15 ` Dmitry Gutov @ 2015-06-08 1:59 ` Stephen Leake 2015-06-09 22:31 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-06-08 1:59 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 06/05/2015 01:08 PM, Stephen Leake wrote: > >> No problem; make sure that directory is in project-source-directories. > > How would that work, in technical terms? Which entity will "make sure" > of that? The user; they are the only ones that know what "the project" is. > Will a project-backend "source directories" implementation > always cl-call-next-method? I don't understand this. > Will the default impl simply return the value of some variable, > assigned by a major mode? default implementation of what function? >> README is a perfectly valid source file; it is written in the "plain >> text" language (for which no compiler is needed ;). > > I see a few problems with that: > > - README has no source code (it doesn't affect the logic of any > program). It's the other way around; there is no object code corresponding to the README source code. But that's not very significant. > "sources" already has a common meaning, What is that? I take it to mean "any human-readable file". Which thus includes README, Makefile, .git/config, etc. > and at least in one proprietary editor it's just one a subset of all > project files (see the nomenclature here: > https://www.jetbrains.com/idea/help/content-root.html). That gives me 404 > Further, if we can separate them, we could make sure > xref-find-references only searches in the "proper" source directories. What do you mean by "proper"? We need it to search what the user wants to search; the user needs to be able to specify that clearly and easily. > - There's no direct way to find the directory holding the README when > you only have the value of load-path or compilation-search-path If the user specifies compilation-search-path (directly or indirectly), and the user wants to be able to search README, then the user must include the directory containing README in compilation-search-path. Is that a problem? > So it would make sense to have a different name (and a different > function/slot/etc) for directories that might be reached that way. I don't follow; can you give more details, and a clear example? >> I have no problem with projects including more than one language in the >> sources; most of mine have Ada, Texinfo, LaTeX, text, elisp, and make >> source code. Which is one reason why project and xref cannot be merged >> (xref must be language-specific, even for tags). > > xref should take the language into account, but it should work across > them as well. For instance, a Ruby method call in an *.html.erb > template file should get recognized as a reference to that method, and > its "find definition" should lead to the corresponding *.rb file. Ah. Now you are talking about one file that mixes source in two languages. Yes, that is a problem. There have been discussions here of multi-mode files. I don't know how that would work in practice. I have projects that mix Ada and C++; the project level cross reference facility handles both, using the current xref API. That works because gcc generates the same cross reference info for those two langauages; I doubt there is something similar for Ruby and html. >> It might make sense to parameterize project-source-directories >> with a language (or major-mode) name, to get the corresponding subset. > > That's probably not necessary: after all, only the default > xref-find-references implementation will use > project-source-directories in a naive way. A smarter xref backend > should determine itself which directories to search. I don't see how that is at all possible, especially in the face of hierarchical projects. Only the user knows what a "project" is; they write the hierachical project files that tell the system what the project structure is. >> In addition, there might be a directory under project root that should >> _not_ be searched; the object file directory for ada-mode, for example. > > project-ignore-directories should be a thing. Probably in the format > of grep-find-ignored-directories (and we might have a function > corresponding to grep-find-ignored-files). Trouble is, > projectile-ignored-directories seems to be more flexible already, so > maybe that will have to be accompanied with an upgrade to how > grep-find-ignored-directories are handled. > >> We can make it a requirement in order to use the general tool. But first >> we have to justify it; ada-mode has never needed that notion; neither >> has elisp. In my experience, only config management needs it. > > Like mentioned above, for example, the distinction would help with the > default xref-find-references implementation. You seem to be missing my point; I see no need for project-root, outside of configuration management. So I don't see why xref should care about "root" at all. >>> On the other hand, after the project root is determined, it might want >>> to add some new element(s) to the source directories list. >> >> "it" is what here ? Can you give an example? > > Some package that defines that root-finding logic. Not every major > mode knows where the relevant source directories are, or it might not > know some of them. As I've said, none of the languages I deal with care about a "root" directory. Only git, mtn, cvs care. Hmm. Java imposes something like a root by forcing the directory structure to match the hiearchical package naming. But referencing other package hierarchies complicates that, just as in Ada and C++. > If we've opened one of the files in a third-party Elisp project that > hasn't been loaded yet (so not in load-path), a project implementation > can still add an element or several to project-source-directories, so > that xref-find-references returns somewhat useful results anyway. What does that have to do with "root"? The third-party elisp file will presumably have some paths in "(require ...)" forms that will define a set of directories to be added to project-source-directories. > Or, say, for Ruby: while the major mode might conceivably be taught > about paths where to find system-wide libraries, the locations of > source files in a given project depend on the project's type, and that > must be something a project implementation knows better. Even if it'll > simply add the root to project-source-directories. What "root"? Is that the directory containing the Ruby file that we opened? Why is that a "root" as opposed to just some source directory? -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-08 1:59 ` Stephen Leake @ 2015-06-09 22:31 ` Dmitry Gutov 2015-06-10 7:13 ` Steinar Bang 2015-07-08 0:25 ` Dmitry Gutov 0 siblings, 2 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-06-09 22:31 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 06/08/2015 04:59 AM, Stephen Leake wrote: > The user; they are the only ones that know what "the project" is. Let's cut out the user out of the discussion and just say that the project system knows stuff. Because it can know everything we want to ask the user; we just need to figure out the right set of questions. >> Will a project-backend "source directories" implementation >> always cl-call-next-method? > > I don't understand this. If project-source-directories is a generic method, maybe we can say that its implementation should almost always (unless it really knows what it's doing) call the next applicable implementation. That implementation could be set up to dispatch based on the value of major-mode. >> Will the default impl simply return the value of some variable, >> assigned by a major mode? > > default implementation of what function? project-source-directories. >> "sources" already has a common meaning, > > What is that? I take it to mean "any human-readable file". Which thus > includes README, Makefile, .git/config, etc. Maybe there's value in allowing this semantic split to (sometimes?) go along different lines, but the two main directions I want to see commands work are these: - Search all files that belong to any of the projects we're currently interested in. - Search only some of those files; ones that are likely programmatically (semantically?) related to the one we're editing right now. The latter could have a decent-enough implementation using Grep, but it would be better if we could only search directories that are related. Maybe it's indeed an excessive requirement, though. After all, backends can provide their own implementations of "find references" already. >> and at least in one proprietary editor it's just one a subset of all >> project files (see the nomenclature here: >> https://www.jetbrains.com/idea/help/content-root.html). > > That gives me 404 That link still works for me. Maybe your email client mangles it somehow? > We need it to search what the user wants to search; the user needs to be > able to specify that clearly and easily. I intend to allow the user to specify the exact directory for xref-find-regexp to search, when called with a prefix. However, allowing to input an arbitrary set of directories is obviously non-trivial (if we want to have an interface that's not annoying). > If the user specifies compilation-search-path (directly or indirectly), > and the user wants to be able to search README, then the user must > include the directory containing README in compilation-search-path. > > Is that a problem? I mentioned compilation-search-path only as an example, though I didn't expect that you might suggest putting the project root into compilation-search-path. Anyway, emacs-lisp-mode doesn't set compilation-search-path. And hopefully you're not suggesting to put project roots into load-path, just to be able to search READMEs. >> So it would make sense to have a different name (and a different >> function/slot/etc) for directories that might be reached that way. > > I don't follow; can you give more details, and a clear example? If my java project is in ~/vc/fooberz, then project-directories would be '("~/vc/fooberz"), and project-source-directories would be '("~/vc/fooberz/src"). That seems obvious enough, so maybe I don't understand the question. > Ah. Now you are talking about one file that mixes source in two > languages. Yes, that is a problem. There have been discussions here of > multi-mode files. It's targentially related, but the difficulties in displaying and editing multi-mode files are entirely different from the subject of this discussion. > I have projects that mix Ada and C++; the project level cross reference > facility handles both, using the current xref API. That works because > gcc generates the same cross reference info for those two langauages; I > doubt there is something similar for Ruby and html. Maybe, or maybe not; but that's not very relevant. If the tool *can* be written, the API is manageable. In the meantime, we can use Grep. >> That's probably not necessary: after all, only the default >> xref-find-references implementation will use >> project-source-directories in a naive way. A smarter xref backend >> should determine itself which directories to search. > > I don't see how that is at all possible, especially in the face of > hierarchical projects. I suppose all parts of the "hierarchical project" will employ the same project definition logic, possibly enabled by the same minor mode. And that minor mode might as well implement the specialized xref backend that will know where to search. > Only the user knows what a "project" is; they write the hierachical > project files that tell the system what the project structure is. See the beginning of this email. > You seem to be missing my point; I see no need for project-root, outside > of configuration management. So I don't see why xref should care about > "root" at all. Or you're missing mine. The distinction I'm (or was) suggesting is not between the root and everything else, but between "source directories" and "other directories the user might want to search in", and we'll often want to see the project's root directory inside the latter list (when not in both). >>>> On the other hand, after the project root is determined, it might want >>>> to add some new element(s) to the source directories list. >>> >>> "it" is what here ? Can you give an example? >> >> Some package that defines that root-finding logic. Not every major >> mode knows where the relevant source directories are, or it might not >> know some of them. > > As I've said, none of the languages I deal with care about a "root" > directory. Only git, mtn, cvs care. Allow me to repeat: "it" is the package that defines the root-finding logic. Defines the logic that determines the bounds of the current (and possibly linked) project(s). The act of find the project root directory may sometimes be skipped in some odd cases, but otherwise, it'll usually be the first step to determining the current project. > Hmm. Java imposes something like a root by forcing the directory > structure to match the hiearchical package naming. But referencing other > package hierarchies complicates that, just as in Ada and C++. So there'll be several roots. Or project-directories. No big deal. >> If we've opened one of the files in a third-party Elisp project that >> hasn't been loaded yet (so not in load-path), a project implementation >> can still add an element or several to project-source-directories, so >> that xref-find-references returns somewhat useful results anyway. > > What does that have to do with "root"? The root directory usually, a) contains data about the project structure, b) includes files we want to search. > The third-party elisp file will presumably have some paths in "(require > ...)" forms that will define a set of directories to be added to > project-source-directories. First of all, those forms specify path relative to *any* load-path element. We can't add every permutation. Second, I was describing the case where the requires load-path element weren't even set yet. Third, either way it won't include the files that depend on the current one. Those are important, too. >> Or, say, for Ruby: while the major mode might conceivably be taught >> about paths where to find system-wide libraries, the locations of >> source files in a given project depend on the project's type, and that >> must be something a project implementation knows better. Even if it'll >> simply add the root to project-source-directories. > > What "root"? Is that the directory containing the Ruby file that we > opened? It's some parent directory of the currently opened file. Probably not the direct parent. It's important because it contains files related to the current one. > Why is that a "root" as opposed to just some source directory? That's a weird question. It's a root of the project. It *is* also one of the source directories, in this example. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-09 22:31 ` Dmitry Gutov @ 2015-06-10 7:13 ` Steinar Bang 2015-07-08 0:25 ` Dmitry Gutov 1 sibling, 0 replies; 87+ messages in thread From: Steinar Bang @ 2015-06-10 7:13 UTC (permalink / raw) To: emacs-devel >>>>> Dmitry Gutov <dgutov@yandex.ru>: > On 06/08/2015 04:59 AM, Stephen Leake wrote: >>> and at least in one proprietary editor it's just one a subset of all >>> project files (see the nomenclature here: >>> https://www.jetbrains.com/idea/help/content-root.html). >> That gives me 404 Could be the ")" or ")." at the end of the URL...? It's always good to use a space after the end of the URL (Gnus managed to extract the URL without the trailing ").", though...). ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-09 22:31 ` Dmitry Gutov 2015-06-10 7:13 ` Steinar Bang @ 2015-07-08 0:25 ` Dmitry Gutov 2015-07-11 13:43 ` Bozhidar Batsov 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-08 0:25 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 06/10/2015 01:31 AM, Dmitry Gutov wrote: > If project-source-directories is a generic method, maybe we can say that > its implementation should almost always (unless it really knows what > it's doing) call the next applicable implementation. > > That implementation could be set up to dispatch based on the value of > major-mode. I've pushed that to the branch 'scratch/project', please take a look. The split between "project directories" and "project source directories" is not 100% necessary, but I think it can be useful, provided the semantic difference between the two is clear. What I want to do with project-source-directories here, is to allow a major mode to provide a default list (or, actually, a function that would compute it, because `load-path' can change at runtime). This way, unless a project implementation explicitly overrides it, Elisp authors will have the whole load-path used for searching. Maybe using a specialized implementation using &context is not the best thing to do there. Alternatively, we can introduce a new variable (like `project-source-directories-function'), and refer to it in the default `project-source-directories' implementation, as well as its docstring. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-08 0:25 ` Dmitry Gutov @ 2015-07-11 13:43 ` Bozhidar Batsov 2015-07-11 14:17 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Bozhidar Batsov @ 2015-07-11 13:43 UTC (permalink / raw) To: Dmitry Gutov; +Cc: Stephen Leake, emacs-devel [-- Attachment #1: Type: text/plain, Size: 1824 bytes --] I finally took a quick look at the current code (and the discussion so far) and I have a few remarks: * even within source directories it's useful to be able to exclude certain subdirs from project-level operations. How is this going to be handled? * it's not 100% clear to me what are major mode writes supposed to provide as "project implementations" * In 24.4 a `vc-root-dir` was added. Seems it overlaps a bit with `project.el` * how can Projectile (or a similar package) leverage `project.el`? On 8 July 2015 at 03:25, Dmitry Gutov <dgutov@yandex.ru> wrote: > On 06/10/2015 01:31 AM, Dmitry Gutov wrote: > > If project-source-directories is a generic method, maybe we can say that >> its implementation should almost always (unless it really knows what >> it's doing) call the next applicable implementation. >> >> That implementation could be set up to dispatch based on the value of >> major-mode. >> > > I've pushed that to the branch 'scratch/project', please take a look. > > The split between "project directories" and "project source directories" > is not 100% necessary, but I think it can be useful, provided the semantic > difference between the two is clear. > > What I want to do with project-source-directories here, is to allow a > major mode to provide a default list (or, actually, a function that would > compute it, because `load-path' can change at runtime). > > This way, unless a project implementation explicitly overrides it, Elisp > authors will have the whole load-path used for searching. > > Maybe using a specialized implementation using &context is not the best > thing to do there. Alternatively, we can introduce a new variable (like > `project-source-directories-function'), and refer to it in the default > `project-source-directories' implementation, as well as its docstring. > > [-- Attachment #2: Type: text/html, Size: 2418 bytes --] ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-11 13:43 ` Bozhidar Batsov @ 2015-07-11 14:17 ` Dmitry Gutov 2015-07-12 14:42 ` Dmitry Gutov 2015-07-13 8:49 ` Bozhidar Batsov 0 siblings, 2 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-11 14:17 UTC (permalink / raw) To: Bozhidar Batsov; +Cc: Stephen Leake, emacs-devel On 07/11/2015 04:43 PM, Bozhidar Batsov wrote: > I finally took a quick look at the current code (and the discussion so > far) and I have a few remarks: > > * even within source directories it's useful to be able to exclude > certain subdirs from project-level operations. How is this going to be > handled? A new function (or two?) will be needed for that. Is there a particular reason Projectile has both ignored-directories and ignored-files? I'm also undecided on what it (or they) should return. Should that be just a list of shell globs? That would integrate fine with grep-find. It also has both grep-find-ignored-directories and grep-find-ignored-directories. I wonder if that was done for performance. > * it's not 100% clear to me what are major mode writes supposed to > provide as "project implementations" major modes aren't really supposed to. Since the current project is dependent on the directory and not a specific files, it's best left to minor modes. An implementation is a new project-find-functions element and a set of cl-defmethod forms to go with it. Maybe also a cl-struct definition for the project structure, but as you can see from the current project.el contents, you can do without it. > * In 24.4 a `vc-root-dir` was added. Seems it overlaps a bit with > `project.el` Only if the project is based on VC. As you can see project-try-vc uses that backend action, if not the function itself. > * how can Projectile (or a similar package) leverage `project.el`? Do something along these lines: (defun project-try-projectile () (when (projectile-project-root) 'projectile) (add-hook 'project-find-functions #'project-try-projectile) (cl-defmethod project-root (project (eql projectile)) (projectile-project-root)) ...and so on. The first difficulty will be that cl-generic (and hence, cl-defmethod) are unavailable in the previous Emacs versions. Maybe put all this in a separate file and (require ...) it conditionally on the Emacs version. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-11 14:17 ` Dmitry Gutov @ 2015-07-12 14:42 ` Dmitry Gutov 2015-07-13 8:49 ` Bozhidar Batsov 1 sibling, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-12 14:42 UTC (permalink / raw) To: Bozhidar Batsov; +Cc: Stephen Leake, emacs-devel On 07/11/2015 05:17 PM, Dmitry Gutov wrote: > I'm also undecided on what it (or they) should return. Should that be > just a list of shell globs? That would integrate fine with grep-find. I've pushed a single-function glob-based implementation. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-11 14:17 ` Dmitry Gutov 2015-07-12 14:42 ` Dmitry Gutov @ 2015-07-13 8:49 ` Bozhidar Batsov 2015-07-13 10:23 ` Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Bozhidar Batsov @ 2015-07-13 8:49 UTC (permalink / raw) To: Dmitry Gutov; +Cc: Stephen Leake, emacs-devel [-- Attachment #1: Type: text/plain, Size: 2512 bytes --] On 11 July 2015 at 17:17, Dmitry Gutov <dgutov@yandex.ru> wrote: > On 07/11/2015 04:43 PM, Bozhidar Batsov wrote: > >> I finally took a quick look at the current code (and the discussion so >> far) and I have a few remarks: >> >> * even within source directories it's useful to be able to exclude >> certain subdirs from project-level operations. How is this going to be >> handled? >> > > A new function (or two?) will be needed for that. Is there a particular > reason Projectile has both ignored-directories and ignored-files? > > I'm also undecided on what it (or they) should return. Should that be just > a list of shell globs? That would integrate fine with grep-find. > > It also has both grep-find-ignored-directories and > grep-find-ignored-directories. I wonder if that was done for performance. Might have been an oversight on my part. Right now nothing comes to mind and I'll have to consult the source to remember. Originally there was only the option to ignore folders, this much I remember. > > > * it's not 100% clear to me what are major mode writes supposed to >> provide as "project implementations" >> > > major modes aren't really supposed to. Since the current project is > dependent on the directory and not a specific files, it's best left to > minor modes. > e.g. Rails mode or something like this? > > An implementation is a new project-find-functions element and a set of > cl-defmethod forms to go with it. Maybe also a cl-struct definition for the > project structure, but as you can see from the current project.el contents, > you can do without it. > > * In 24.4 a `vc-root-dir` was added. Seems it overlaps a bit with >> `project.el` >> > > Only if the project is based on VC. As you can see project-try-vc uses > that backend action, if not the function itself. > > * how can Projectile (or a similar package) leverage `project.el`? >> > > Do something along these lines: > > (defun project-try-projectile () > (when (projectile-project-root) 'projectile) > > (add-hook 'project-find-functions #'project-try-projectile) > > (cl-defmethod project-root (project (eql projectile)) > (projectile-project-root)) > > ...and so on. > > The first difficulty will be that cl-generic (and hence, cl-defmethod) are > unavailable in the previous Emacs versions. Maybe put all this in a > separate file and (require ...) it conditionally on the Emacs version.' > Yeah, backward compatibility is problematic. I guess those can't be backported to the `cl-lib` package in ELPA? [-- Attachment #2: Type: text/html, Size: 3842 bytes --] ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-13 8:49 ` Bozhidar Batsov @ 2015-07-13 10:23 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-13 10:23 UTC (permalink / raw) To: Bozhidar Batsov; +Cc: Stephen Leake, emacs-devel On 07/13/2015 11:49 AM, Bozhidar Batsov wrote: > Might have been an oversight on my part. Right now nothing comes to mind > and I'll have to consult the source to remember. Originally there was only > the option to ignore folders, this much I remember. I've looked at the code, and you have two functions that returns globs to ignore: projectile-paths-to-ignore and projectile-patterns-to-ignore. They're split along different lines, though (rooted vs non-rooted patterns). I'm not sure how useful that would be. And projectile-ignored-files/directories return actual file names inside the project, that's not what I want here either. For now, it seems you'll implement `project-ignores' by using the value returned by `projectile-parse-dirconfig-file' (with a slight change: replacing / at bos with ./) > e.g. Rails mode or something like this? Yep. ENSIME, Malabar, etc. Or Projectile. > Yeah, backward compatibility is problematic. I guess those can't be > backported to the `cl-lib` package in ELPA? Well, maybe if there's enough of a demand someone will do it. It's not there yet, and I'm not sure how much work it'll require. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-05 10:08 ` Stephen Leake 2015-06-05 13:03 ` Stephen Leake 2015-06-07 23:15 ` Dmitry Gutov @ 2015-07-24 23:43 ` Dmitry Gutov 2015-07-25 0:55 ` Stephen Leake 2 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-24 23:43 UTC (permalink / raw) To: Stephen Leake, emacs-devel Let's maybe continue this discussion? On 06/05/2015 01:08 PM, Stephen Leake wrote: > No, xref-find-regexp should search project-source-directories. > > Most real projects include other projects, so limiting the search to > only the top project is wrong in general (although that might be a > useful option in some use cases). I suppose, and maybe I was wrong to suggest a distinction between "source" directories and the rest of them. However, here's one which could be better: directories to search vs. directories to modify. Like, I might want to search the full system includes path, in a C project, but when performing a replacement operation (such as xref-query-replace), I should probably limit myself to the directories with code I actually own. At the moment, it's the distinction I imagine for project-search-path vs project-directories. > In addition, there might be a directory under project root that should > _not_ be searched; the object file directory for ada-mode, for example. We have that in project-ignores now. > We can make [root] a requirement in order to use the general tool. But first > we have to justify it; ada-mode has never needed that notion; neither > has elisp. In my experience, only config management needs it. > ... > It also points out that "project root" is poorly defined; I think that > is because it's not at all clear when and why we need it. Here's the first use case I know of: There's a third-party project called rspec-mode. It allows one to switch between a source file and its test, and to run tests in a compilation buffer. For both purposes, it needs to determine the directory which to build the relative paths against, and in which to run the 'rspec' program. Currently, it does a locate-dominating-file like routine (looking for Gemfile or Rakefile), but it could switch to the project API in the future. You're probably aware of similar tools. So, now we have both project-root and project-directories. However, the relationship between them looks murky, from the consumer's standpoint. Like, must the former always be included in the latter? I'm considering removing project-root, and copying a part of Atom's API [0]: adding a function project-relativize-path, which will, given an arbitrary directory, if it's inside project-directories, split its file name in two and return a cons, with one of project-directories in the car, and the relative file name against it in the cdr. Then instead of project-root, a consumer will use (car (project-relativize-path default-directory)). Thoughts? [0] https://atom.io/docs/api/v1.0.2/Project#instance-relativizePath ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-24 23:43 ` Dmitry Gutov @ 2015-07-25 0:55 ` Stephen Leake 2015-07-25 7:29 ` Eli Zaretskii 2015-07-26 2:11 ` Dmitry Gutov 0 siblings, 2 replies; 87+ messages in thread From: Stephen Leake @ 2015-07-25 0:55 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > Let's maybe continue this discussion? > > On 06/05/2015 01:08 PM, Stephen Leake wrote: > >> No, xref-find-regexp should search project-source-directories. >> >> Most real projects include other projects, so limiting the search to >> only the top project is wrong in general (although that might be a >> useful option in some use cases). > > I suppose, and maybe I was wrong to suggest a distinction between > "source" directories and the rest of them. However, here's one which > could be better: directories to search vs. directories to modify. > > Like, I might want to search the full system includes path, in a C > project, but when performing a replacement operation (such as > xref-query-replace), I should probably limit myself to the directories > with code I actually own. Yes, that's a good use case. > At the moment, it's the distinction I > imagine for project-search-path vs project-directories. I would suggest the terminology "main project" vs "included projects" for this; the "main project" is the code you own; the "included projects" are all the others (included the system includes). So this would be an argument to xref-find-regexp; (defun xref-find-regexp (regexp &optional not-included-projects) "Search all files in all project directories for REGEXP. If NOT-INCLUDED-PROJECTS is non-nil (default nil), search only the main project. Otherwise, search the main project and included projects." To implement this, project-search-path might want to be an alist of: (PROJECT-NAME <search-path>) where the first element is the main project and the rest are the included ones. >> In addition, there might be a directory under project root that should >> _not_ be searched; the object file directory for ada-mode, for example. > > We have that in project-ignores now. Right. That's not consistent with current things like load-path. I should think at some point in the implementation of any "search project" function you will want an explicit list of directories (and maybe files) to search, which will be the union of project-search-path minus project-ignores. So it seems simpler to specify that list directly. The user interface (the syntax of the project file) could be structured as includes and ignores (excludes). >> We can make [root] a requirement in order to use the general tool. But first >> we have to justify it; ada-mode has never needed that notion; neither >> has elisp. In my experience, only config management needs it. >> ... >> It also points out that "project root" is poorly defined; I think that >> is because it's not at all clear when and why we need it. > > Here's the first use case I know of: > > There's a third-party project called rspec-mode. It allows one to > switch between a source file and its test, and to run tests in a > compilation buffer. For both purposes, it needs to determine the > directory which to build the relative paths against, and in which to > run the 'rspec' program. Currently, it does a locate-dominating-file > like routine (looking for Gemfile or Rakefile), but it could switch to > the project API in the future. Ok, makes sense. > You're probably aware of similar tools. monotone, git, subversion ... :). Recently I've been playing with Google's Android Studio (shudder; only to make Emacs do what it does, I promise :). It defines a project root by a dominating file. On the other hand, AdaCore GPS (an Ada-specific IDE) defines a project by a project file that specifies various lists of directories (source, object, executable); there is no root directory. The project file itself can be anywhere, it doesn't have to be in one of the listed directories. That's what Emacs Ada mode does as well. > So, now we have both project-root and project-directories. However, > the relationship between them looks murky, from the consumer's > standpoint. Like, must the former always be included in the latter? I think they are disjoint concepts. Nothing wrong with keeping both; some tools will need them, some won't. > I'm considering removing project-root, and copying a part of Atom's > API [0]: adding a function project-relativize-path, which will, given > an arbitrary directory, if it's inside project-directories, split its > file name in two and return a cons, with one of project-directories in > the car, and the relative file name against it in the cdr. If project-directories contains all directories (not just root ones), this is just (file-name-directory path) . (file-name-non-directory path). > Then instead of project-root, a consumer will use (car > (project-relativize-path default-directory)). I don't follow; that means the value of 'root directory' changes depending on where the current file is. Unless project-directories contains only one directory, which would then be the root. Given the variable project-root, it makes sense to define project-relativize-path to return the portion of the path that is relative to project-root, and error if there is no such portion (ie an absolute path outside the root tree). Monotone has such a function in the C++ code. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-25 0:55 ` Stephen Leake @ 2015-07-25 7:29 ` Eli Zaretskii 2015-07-26 2:12 ` Dmitry Gutov 2015-07-26 2:11 ` Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Eli Zaretskii @ 2015-07-25 7:29 UTC (permalink / raw) To: Stephen Leake; +Cc: emacs-devel > From: Stephen Leake <stephen_leake@stephe-leake.org> > Date: Fri, 24 Jul 2015 19:55:09 -0500 > > I would suggest the terminology "main project" vs "included projects" > for this; the "main project" is the code you own; the "included > projects" are all the others (included the system includes). IMO, system header files are not the same as "included projects", because the system headers are immutable. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-25 7:29 ` Eli Zaretskii @ 2015-07-26 2:12 ` Dmitry Gutov 2015-07-26 2:45 ` Eli Zaretskii 2015-07-26 11:25 ` Stephen Leake 0 siblings, 2 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-26 2:12 UTC (permalink / raw) To: Eli Zaretskii, Stephen Leake; +Cc: emacs-devel On 07/25/2015 10:29 AM, Eli Zaretskii wrote: > IMO, system header files are not the same as "included projects", > because the system headers are immutable. AFAICT, that's exactly what Stephen meant by "included projects": projects, whose code we're not allowed to change. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 2:12 ` Dmitry Gutov @ 2015-07-26 2:45 ` Eli Zaretskii 2015-07-26 11:25 ` Stephen Leake 1 sibling, 0 replies; 87+ messages in thread From: Eli Zaretskii @ 2015-07-26 2:45 UTC (permalink / raw) To: Dmitry Gutov; +Cc: stephen_leake, emacs-devel > From: Dmitry Gutov <dgutov@yandex.ru> > Date: Sun, 26 Jul 2015 05:12:44 +0300 > Cc: emacs-devel@gnu.org > > On 07/25/2015 10:29 AM, Eli Zaretskii wrote: > > > IMO, system header files are not the same as "included projects", > > because the system headers are immutable. > > AFAICT, that's exactly what Stephen meant by "included projects": > projects, whose code we're not allowed to change. Then the name is misleading. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 2:12 ` Dmitry Gutov 2015-07-26 2:45 ` Eli Zaretskii @ 2015-07-26 11:25 ` Stephen Leake 1 sibling, 0 replies; 87+ messages in thread From: Stephen Leake @ 2015-07-26 11:25 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/25/2015 10:29 AM, Eli Zaretskii wrote: > >> IMO, system header files are not the same as "included projects", >> because the system headers are immutable. > > AFAICT, that's exactly what Stephen meant by "included projects": > projects, whose code we're not allowed to change. No, _some_ included projects are read-only; others are read-write. I don't think the distinction is useful at the project level; file permissions handle it. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-25 0:55 ` Stephen Leake 2015-07-25 7:29 ` Eli Zaretskii @ 2015-07-26 2:11 ` Dmitry Gutov 2015-07-26 11:22 ` Stephen Leake 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-26 2:11 UTC (permalink / raw) To: emacs-devel Hi Stephen, Sorry for the delay; I had to think on it. On 07/25/2015 03:55 AM, Stephen Leake wrote: > I would suggest the terminology "main project" vs "included projects" > for this; the "main project" is the code you own; the "included > projects" are all the others (included the system includes). There's sense in this, but is "an included project you don't own, which is not in system includes" a significant use case? In my own experience, either the projects on which the current one depends are installed in system directories, and thus are indistinguishable from system includes (such as Ruby gems, installed via "gem install"), or the linked project's tree lives near the current one, and we can consider it a part of the main project, for the purposes of editing. Because even if the second project is a third-party dependency, sometimes you want to make changes to it anyway, and the user might get miffed if xref-query-replace prohibits that. > So this would be an argument to xref-find-regexp; > > (defun xref-find-regexp (regexp &optional not-included-projects) > "Search all files in all project directories for REGEXP. > If NOT-INCLUDED-PROJECTS is non-nil (default nil), search only the main > project. Otherwise, search the main project and included projects." It should probably be a new command, because otherwise specifying that second argument interactively would be a pain (C-u already makes xref-find-regexp ask a bunch of questions). > To implement this, project-search-path might want to be an alist of: > > (PROJECT-NAME <search-path>) > > where the first element is the main project and the rest are the > included ones. Maybe do it differently? If we do have linked projects (and the separation between main/included), some generic function, probably named differently, would return a list of those project instances. Then the caller will ask each instance for its root (or the directories). There's no need for the main project's search-path entry there, xref-find-regexp can just use its root (or the directories) as well. >> We have that in project-ignores now. > > Right. That's not consistent with current things like load-path. I Indeed. But maybe supporting ignores for project-search-path is not a big priority. There's not much to ignore in /usr/include, for instance. The directories in load-path also usually don't contain weird files or build artefacts (and when they do, it's not like elisp-search-path really can account for those). So maybe a good first approximation would be to simply not apply the ignores list to the project-search-path elements outside of the project tree. If that's not good enough, project-ignores can accept a directory as its argument (it must be among those returned by project-directories, or project-search-path), and return the appropriate ignores list. This will definitely help if each element in project-directories is a separate (e.g. Git) repository, with its own .gitignore. > should think at some point in the implementation of any "search project" > function you will want an explicit list of directories (and maybe files) > to search, which will be the union of project-search-path minus > project-ignores. So it seems simpler to specify that list directly. I'm not sure about that. It seems then the implementation will have to walk the whole project tree itself, and collect all directories that don't match the ignores (because .gitignore syntax allows not-anchored entries for directories). And if we're to support ignoring files (and obvious use case), it'll become a list of files, which can grow pretty big. I think it's better to delegate this job to 'find', as long as that doesn't force us to compromise on functionality. > The user interface (the syntax of the project file) could be structured > as includes and ignores (excludes). .gitignore also supports whitelisting entries (which override the excludes). I haven't gotten around to writing support for those. >> You're probably aware of similar tools. > > monotone, git, subversion ... :). Not sure about the other two, but Git doesn't need to be told the project root. It can be called from any directory, and works with relative paths quite well. > Recently I've been playing with Google's Android Studio (shudder; only > to make Emacs do what it does, I promise :). It defines a project root > by a dominating file. Of course. Why else one would be playing with it? ;) In this case, project-directories can return a list with one element. > On the other hand, AdaCore GPS (an Ada-specific IDE) defines a project > by a project file that specifies various lists of directories (source, > object, executable); there is no root directory. The project file itself > can be anywhere, it doesn't have to be in one of the listed directories. > That's what Emacs Ada mode does as well. In this case, it seems, the relevant project implementation will be based on user explicitly telling it where the project file lives (by calling a particular command, with a prompt), and then the project will be enabled globally, for the whole Emacs session. project-directories then will return some of the directories specified in the project file (except the object ones, maybe, if there's nothing we can edit there), but probably (?) not the one where the project file lies, if it's somewhere separately. Would there be any significant value in project-root returning the directory of the project file? > I think they are disjoint concepts. Nothing wrong with keeping both; > some tools will need them, some won't. That would mean documenting the distinction, so that implementers know what to do. I'm increasingly less clear on it. > If project-directories contains all directories (not just root ones), > this is just (file-name-directory path) . (file-name-non-directory > path). It would be. But I expect project-directories to only include the roots. IOW, no project-directories element should be a subdirectory of another. >> Then instead of project-root, a consumer will use (car >> (project-relativize-path default-directory)). > > I don't follow; that means the value of 'root directory' changes > depending on where the current file is. That's correct. If the file is inside the "main" project directory, it will use the main project root. If the file is in another directory, it's in a "linked" body of code (which would often be called a project as well), and its root is used. Atom has taken this approach: each element in atom.project.getDirectories() corresponds to a Git repository. See the link I sent previously, and also https://github.com/atom/atom/pull/3691. This may look wrong for certain configurations (your AdaCore example comes to mind), but would the concept of root be even useful in that case? > Given the variable project-root, it makes sense to define > project-relativize-path to return the portion of the path that is > relative to project-root, and error if there is no such portion (ie an > absolute path outside the root tree). Monotone has such a function in > the C++ code. Sure. But if project-root is not among project-directories, I'm not sure when, and for what cases, that directory can be used. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 2:11 ` Dmitry Gutov @ 2015-07-26 11:22 ` Stephen Leake 2015-07-26 17:23 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-07-26 11:22 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > Hi Stephen, > > Sorry for the delay; I had to think on it. > > On 07/25/2015 03:55 AM, Stephen Leake wrote: > >> I would suggest the terminology "main project" vs "included projects" >> for this; the "main project" is the code you own; the "included >> projects" are all the others (included the system includes). > > There's sense in this, but is "an included project you don't own, > which is not in system includes" a significant use case? Yes; any third party library that you should not edit, because the sources are controlled upstream. > In my own experience, either the projects on which the current one > depends are installed in system directories, and thus are > indistinguishable from system includes (such as Ruby gems, installed > via "gem install"), Ok. I don't see the notion of "system directories" as important here; the important point is whether the libraries are read-only or read/write. We should not rely on where they happen to reside to decide that; sometimes I edit "system directories" when working on patches for upstream. Or just ignore the read-only issue in the project code; file permissions are good enough. >> should think at some point in the implementation of any "search project" >> function you will want an explicit list of directories (and maybe files) >> to search, which will be the union of project-search-path minus >> project-ignores. So it seems simpler to specify that list directly. > > I'm not sure about that. It seems then the implementation will have to > walk the whole project tree itself, and collect all directories that > don't match the ignores (because .gitignore syntax allows not-anchored > entries for directories). Yes. You do that once when the project is opened, and cache the result in the variable project-search-path. Then all code that searches projects is simpler; it deals with a simple list of directories, plus a list of file extensions (just as locate-file does). >> The user interface (the syntax of the project file) could be structured >> as includes and ignores (excludes). > > .gitignore also supports whitelisting entries (which override the > excludes). I haven't gotten around to writing support for those. Right. .gitignore is part of "the project file" in this sense. >> Recently I've been playing with Google's Android Studio (shudder; only >> to make Emacs do what it does, I promise :). It defines a project root >> by a dominating file. > > Of course. Why else one would be playing with it? ;) > > In this case, project-directories can return a list with one element. No, Emacs needs to read the Studio project file and extract the search path; it's just another project file syntax. >> On the other hand, AdaCore GPS (an Ada-specific IDE) defines a project >> by a project file that specifies various lists of directories (source, >> object, executable); there is no root directory. The project file itself >> can be anywhere, it doesn't have to be in one of the listed directories. >> That's what Emacs Ada mode does as well. > > In this case, it seems, the relevant project implementation will be > based on user explicitly telling it where the project file lives (by > calling a particular command, with a prompt), and then the project > will be enabled globally, for the whole Emacs session. Yes. No different from any other Emacs project; the user has to tell Emacs where it is. If that's a directory, Emacs searches for project file extensions it knows about, and reads that. If it's a file, no searching needed. > Would there be any significant value in project-root returning the > directory of the project file? Emacs should trust what the project file says; it has an explicit list of "source code directories". Emacs should not care whether that contains the project file itself or not; that's up to the user. Since the external tools that use the project file can report errors in its syntax, I usually include the project file in a directory that is in the source path. But that's my choice, not Emacs's. >> I think they are disjoint concepts. Nothing wrong with keeping both; >> some tools will need them, some won't. > > That would mean documenting the distinction, so that implementers know > what to do. I'm increasingly less clear on it. project-root; a single absolute directory, usually containing significant files for external tools (ie .git/), often the root of the source tree. note that the _only_ requirement here is that it is a single absolute directory; the rest is just suggestions. project-search-path; a list of directories, containing files related to the project. This will normally include project-root. It may also contain included projects. this might want to be split into project-source-search-path, project-object-search-path, but since Emacs is mostly interested in editing sources, only the source path is interesting. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 11:22 ` Stephen Leake @ 2015-07-26 17:23 ` Dmitry Gutov 2015-07-26 18:57 ` Stephen Leake 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-26 17:23 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/26/2015 02:22 PM, Stephen Leake wrote: > Yes; any third party library that you should not edit, because the > sources are controlled upstream. Is it common to have its sources checked out in a directory nearby? > Ok. I don't see the notion of "system directories" as important here; > the important point is whether the libraries are read-only or > read/write. We should not rely on where they happen to reside to decide > that; sometimes I edit "system directories" when working on patches for > upstream. There is a difference, from my perspective: usually, there are no object files in system libraries, no build files, or .gitignore files. > Or just ignore the read-only issue in the project code; file permissions > are good enough. Maybe ignoring that is fine (are we backtracking on the separation of the directories for the purposes of search and edit operations?), but file permissions won't help in a lot of cases. In Ruby and Python communities, for instance, it's common to install the language runtime, with all libraries, inside the developer's home directory (using RVM/rbenv for Ruby, or virtualenv for Python). And I might edit some of files there for debugging purposes, but if I forget about those edits later, only bad can come of it. Likewise, I usually wouldn't want a global replace operation to touch them. >> I'm not sure about that. It seems then the implementation will have to >> walk the whole project tree itself, and collect all directories that >> don't match the ignores (because .gitignore syntax allows not-anchored >> entries for directories). > > Yes. You do that once when the project is opened, and cache the result And then the user creates a new dir (or switches to a different branch, revision, etc), and the cache goes out of sync. How do we handle that? On the other hand, delegating the scan of the filesystem to 'find' seems performant enough. Someone correct me if I'm wrong, but most of the time is spent in Grep anyway. > in the variable project-search-path. Then all code that searches > projects is simpler; it deals with a simple list of directories, plus a > list of file extensions (just as locate-file does). locate-file seems to use a whitelist of suffixes, which is distinctly less powerful than with what's offered by .gitignore. I think keeping compatibility with it (and similar .bzrignore and .hgignore) is valuable. > No, Emacs needs to read the Studio project file and extract the search > path; it's just another project file syntax. But according to you, the "search path" will be just the directories inside the project's tree, right? That can be determined solely on the location of that file. What will "included projects" look like, in an average Android project? > Yes. No different from any other Emacs project; the user has to tell > Emacs where it is. If that's a directory, Emacs searches for project > file extensions it knows about, and reads that. If it's a file, no > searching needed. The VC project implementation is currently based on the current buffer's directory. No need to explicitly specify it, Emacs find the repo root without user intervention. If you've ever tried Projectile, it works in a similar fashion. Android projects could be detected like that as well. > Emacs should trust what the project file says; it has an explicit list of > "source code directories". Emacs should not care whether that contains > the project file itself or not; that's up to the user. Will this miss all the other directories in the project, such as documentation, maybe? One would expect xref-find-regexp to search them, too. > Since the external tools that use the project file can report errors in > its syntax, I usually include the project file in a directory that is in > the source path. But that's my choice, not Emacs's. One would expect xref-find-regexp to search in the project file, too. > project-root; a single absolute directory, usually containing > significant files for external tools (ie .git/), often the root of the > source tree. Would we have project-root mainly for the purpose of xref-find-regexp being able to search in the project file? If project-root is not in project-search-path, can there be other, unrelated files in that directory? How would we make sure xref-find-regexp ignores them? Seems like any other tool that would use it would be specific to that kind of project (such as project file checker). Maybe the path to the project file should be set via metadata, using a key like ada-project-file. > note that the _only_ requirement here is that it is a single > absolute directory; the rest is just suggestions. That's a problem, IMO. If project-root is loosely defined, it's not useful to third-party code that doesn't know much more about your kind of project. And as per below, we're be including source directories from both the current project and the included projects into project-search-path, right? Seemingly on equal rights. But then, each of those projects likely has a .gitignore, a build file, and so on, but project-root will only return the location of project files for the current project? That seems inconsistent. > project-search-path; a list of directories, containing files related to > the project. This will normally include project-root. It may also > contain included projects. So, this tries to combine project-search-path and project-directories, as they're defined now? From the previous email, I imagined a project-specific project-search-path plus an accessor like project-dependencies, which would contain a list of project objects corresponding to the included projects. > this might want to be split into project-source-search-path, > project-object-search-path, but since Emacs is mostly interested in > editing sources, only the source path is interesting. Right. I'd call it project-directories, though. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 17:23 ` Dmitry Gutov @ 2015-07-26 18:57 ` Stephen Leake 2015-07-26 23:56 ` John Yates 2015-07-27 13:00 ` Dmitry Gutov 0 siblings, 2 replies; 87+ messages in thread From: Stephen Leake @ 2015-07-26 18:57 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > In Ruby and Python communities, for instance, it's common to install > the language runtime, with all libraries, inside the developer's home > directory (using RVM/rbenv for Ruby, or virtualenv for Python). And I > might edit some of files there for debugging purposes, but if I forget > about those edits later, only bad can come of it. Likewise, I usually > wouldn't want a global replace operation to touch them. Then it's not "global", is it? It's "operate on this set of projects, exclude that set". So you need a way to specify sets of projects. >>> I'm not sure about that. It seems then the implementation will have to >>> walk the whole project tree itself, and collect all directories that >>> don't match the ignores (because .gitignore syntax allows not-anchored >>> entries for directories). >> >> Yes. You do that once when the project is opened, and cache the result > > And then the user creates a new dir (or switches to a different > branch, revision, etc), and the cache goes out of sync. How do we > handle that? project-cache-refresh. Only the user knows when it should be done; they may want to keep the slightly out of date cache because something isn't finished with the edits yet. >> No, Emacs needs to read the Studio project file and extract the search >> path; it's just another project file syntax. > > But according to you, the "search path" will be just the directories > inside the project's tree, right? No, I have been explicitly saying that is not the case; the search path is whatever the project file says it is. > What will "included projects" look like, in an average Android > project? A list of project names. Depending on the project manager, that implies a list of project files in one syntax or another, each of which specifies a set of project directories (explicitly or implicitly). >> Yes. No different from any other Emacs project; the user has to tell >> Emacs where it is. If that's a directory, Emacs searches for project >> file extensions it knows about, and reads that. If it's a file, no >> searching needed. > > The VC project implementation is currently based on the current > buffer's directory. No need to explicitly specify it, Emacs find the > repo root without user intervention. Yes. > If you've ever tried Projectile, it works in a similar fashion. > Android projects could be detected like that as well. Maybe. Depends on the project manager software. Some have a dominating file, some do it some other way. Why do we care? Just support whatever method the project manger backend requires. project-find-project-file is a dispatching function on the backend. >> Emacs should trust what the project file says; it has an explicit list of >> "source code directories". Emacs should not care whether that contains >> the project file itself or not; that's up to the user. > > Will this miss all the other directories in the project, such as > documentation, maybe? One would expect xref-find-regexp to search > them, too. Trust what the project file says. The user told Emacs to use the project file; it should not try to be smarter. If the user wants documentation in the project, then it is in the project file. If the project manager tool is brain-dead and can't handle documentation, the the user can switch tools. >> Since the external tools that use the project file can report errors in >> its syntax, I usually include the project file in a directory that is in >> the source path. But that's my choice, not Emacs's. > > One would expect xref-find-regexp to search in the project file, too. Only if I tell it to by including it in the project file. If I tell it to _not_ search the project file, it should do that. >> project-root; a single absolute directory, usually containing >> significant files for external tools (ie .git/), often the root of the >> source tree. > > Would we have project-root mainly for the purpose of xref-find-regexp > being able to search in the project file? No. We should have project-root for tools that need a root directory. xref-find-regexp needs a search path, not a root directory. > If project-root is not in project-search-path, can there be other, > unrelated files in that directory? How would we make sure > xref-find-regexp ignores them? xref-find-regexp searches project-search-path. project-search-path is set by reading the project files. The project files are provided by the user. Voila, xref-find-regexp does _exactly_ what the user wants, no more, no less. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 18:57 ` Stephen Leake @ 2015-07-26 23:56 ` John Yates 2015-07-27 1:49 ` Dmitry Gutov 2015-07-27 11:12 ` Stephen Leake 2015-07-27 13:00 ` Dmitry Gutov 1 sibling, 2 replies; 87+ messages in thread From: John Yates @ 2015-07-26 23:56 UTC (permalink / raw) To: Stephen Leake; +Cc: Emacs developers [-- Attachment #1: Type: text/plain, Size: 1640 bytes --] On Sun, Jul 26, 2015 at 2:57 PM, Stephen Leake < stephen_leake@stephe-leake.org> wrote: > > And then the user creates a new dir (or switches to a different > > branch, revision, etc), and the cache goes out of sync. How do we > > handle that? > > project-cache-refresh. Only the user knows when it should be done; they > may want to keep the slightly out of date cache because something isn't > finished with the edits yet. This seems to me to be "rationalizing a zit". I conjecture that you only advocate such a UI because you imagine a not-particularly performant implementation. If there were no cost to keeping the cache current (equivalent to stating that a naive user never would _never_ need to learn of that cache) would you still advocate such a UI? In small projects the cost of a simplistic brute-force change determination is likely to be entirely acceptable. On systems that support emacs' file-notify watching every directory within a project should be able to keep cache refresh overhead to negligible levels (potentially improved by only watch the modifiable portions of the project). In the exceedingly rare case where a user needs to keep work hidden from project searches he can either modify the project definition to blacklist temporarily the areas he wants excluded. Alternatively we could provide a toggle for very advanced users to suppress cache refersh. Bottom line: start by getting the model of state and the UI right. After that if the performance in corner cases needs help then give advanced users tools to tune the system. But please do not foist a user-maintained cache on casual users. /john [-- Attachment #2: Type: text/html, Size: 2190 bytes --] ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 23:56 ` John Yates @ 2015-07-27 1:49 ` Dmitry Gutov 2015-07-27 11:12 ` Stephen Leake 1 sibling, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-27 1:49 UTC (permalink / raw) To: John Yates, Stephen Leake; +Cc: Emacs developers On 07/27/2015 02:56 AM, John Yates wrote: > If there were no cost to keeping the cache > current (equivalent to stating that a naive user never would _never_ > need to learn of that cache) would you still advocate such a UI? I think the word "cache" itself implies a storage that can get stale. > Bottom line: start by getting the model of state and the UI right. > After that if the performance in corner cases needs help then give > advanced users tools to tune the system. But please do not foist a > user-maintained cache on casual users. Right. Even if there's a cache mechanism, I'd very much like to see its invalidation by automatic means (via file-notify, or similar), at least by default. Anyway, I'd prefer to implement as many operations as feasible without needing that cache. If the search through the project tree spends an order of magnitude more time searching than listing the files, there's not point in caching the latter. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 23:56 ` John Yates 2015-07-27 1:49 ` Dmitry Gutov @ 2015-07-27 11:12 ` Stephen Leake 2015-07-27 11:27 ` Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-07-27 11:12 UTC (permalink / raw) To: emacs-devel John Yates <john@yates-sheets.org> writes: > On Sun, Jul 26, 2015 at 2:57 PM, Stephen Leake < > stephen_leake@stephe-leake.org> wrote: > >> > And then the user creates a new dir (or switches to a different >> > branch, revision, etc), and the cache goes out of sync. How do we >> > handle that? >> >> project-cache-refresh. Only the user knows when it should be done; they >> may want to keep the slightly out of date cache because something isn't >> finished with the edits yet. > > > This seems to me to be "rationalizing a zit". I conjecture that you only > advocate such a UI because you imagine a not-particularly performant > implementation. If there were no cost to keeping the cache current > (equivalent to stating that a naive user never would _never_ need to learn > of that cache) would you still advocate such a UI? The use case is this: You have checked out a fully working workspace, in order to do some refactoring. Emacs reads the project file and caches everything. In particular, xref-find-definition works nicely. Now you start to make changes. Some of the files have incorrect syntax, or don't compile because of undefined names. If you attempt to refresh the cache at this point, you will lose _all_ xref, because of the syntax and name errors. On the other hand, the existing cache allows most xref commands to still work. So you keep it for a while. This assumes the mechanism that refreshes the cache is sensitive to such issues. This is true for the Ada xref backend; it uses the xref info output by the compiler, so files must compile. If the file fails to compile for any reason, there is no xref info available. It is also true that the user time penalty for refreshing the cache is significant for Ada mode, and grows with the size of the project. > In small projects the cost of a simplistic brute-force change determination > is likely to be entirely acceptable. Yes, that's true, especially if the cache builder handles syntax errors gracefully, and is very fast. So that should be one option. > In the exceedingly rare case where a user needs to keep work hidden from > project searches he can either modify the project definition to blacklist > temporarily the areas he wants excluded. That's a different use case than I'm talking about, and one I've never run across; can you elaborate? I guess that's one way to implement "find-replace in this subset of the project files". > Alternatively we could provide a toggle for very advanced users to > suppress cache refresh. Yes, that's what I'd like. > Bottom line: start by getting the model of state and the UI right. After > that if the performance in corner cases needs help then give advanced users > tools to tune the system. But please do not foist a user-maintained cache > on casual users. Yes, good approach. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-27 11:12 ` Stephen Leake @ 2015-07-27 11:27 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-27 11:27 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/27/2015 02:12 PM, Stephen Leake wrote: > Emacs reads the project file and caches everything. In particular, > xref-find-definition works nicely. > > Now you start to make changes. Some of the files have incorrect syntax, > or don't compile because of undefined names. > > If you attempt to refresh the cache at this point, you will lose _all_ > xref, because of the syntax and name errors. I thought you were talking about caching the list of files, not their contents? The former won't help you with keeping xref-find-definitions working. The latter is a whole different (complex) discussion, largely unrelated to the project API. We've touched on it in the thread about hidden buffers. > On the other hand, the existing cache allows most xref commands to still > work. So you keep it for a while. You may be thinking of some cached data somewhere in the implementation of the xref backend. That's a valid approach, but neither the project API, nor xref API should know anything about it. It's an implementation detail. > This assumes the mechanism that refreshes the cache is sensitive to such > issues. This is true for the Ada xref backend; it uses the xref info > output by the compiler, so files must compile. If the file fails to > compile for any reason, there is no xref info available. You can keep the xref buffer with the results of the last search, for a while, but as you continue editing files, those entries will also begin to point at wrong locations. The exact point at which they stop working will depend on the specific implementation of xref-location-marker, though. xref-etags-location is designed to be quite resilient, for instance. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-26 18:57 ` Stephen Leake 2015-07-26 23:56 ` John Yates @ 2015-07-27 13:00 ` Dmitry Gutov 2015-07-27 13:02 ` Dmitry Gutov 2015-07-28 1:21 ` Stephen Leake 1 sibling, 2 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-27 13:00 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/26/2015 09:57 PM, Stephen Leake wrote: > Then it's not "global", is it? It's "operate on this set of projects, > exclude that set". So you need a way to specify sets of projects. Maybe the "global" is not the right word, but I'm quite confident that if I ask an average user what a "search across project" action should do, it would replace in the current project, (maybe) in the included ones, and definitely not in the installed libraries. Even if they're writable. Even if we add a "choose which projects to touch" interface, it's good to have a reasonable default. >> But according to you, the "search path" will be just the directories >> inside the project's tree, right? > > No, I have been explicitly saying that is not the case; the search path > is whatever the project file says it is. Sorry, I've replied to that part before reading the rest, and then forgot about it. > A list of project names. Depending on the project manager, that implies > a list of project files in one syntax or another, each of which > specifies a set of project directories (explicitly or implicitly). Could you give an example? Since they're referred only by names, this sounds like a list of dependencies on the installed packages. Those we usually don't want to edit, and I'd classify them as system includes. > Trust what the project file says. The user told Emacs to use the project > file; it should not try to be smarter. So, xref-find-regexp won't call project-root. Got it. > If the user wants documentation in the project, then it is in the > project file. If the project manager tool is brain-dead and can't handle > documentation, the the user can switch tools. In that case, will the user add the documentation directory to the search-path section of the project file? > No. We should have project-root for tools that need a root directory. > xref-find-regexp needs a search path, not a root directory. It could include the root directory in the search path, if it's not already there. But it seems you don't want to do that. Okay. > xref-find-regexp searches project-search-path. project-search-path is > set by reading the project files. The project files are provided by the > user. Voila, xref-find-regexp does _exactly_ what the user wants, no > more, no less. Very well. To sum up, I see a strong case for project-search-path as you described it (maybe under a different name; like project-directories, as it is now), and so far, no real case for project-root. Further, the system search path should be separate, because it's usually specified like that in project files already, in my experience. And if someone were to create, say, an ECB panel showing the current project layout, they might appreciate having the information about these directories being different. If you (or someone else) don't like using "project search path" for the system libraries path, feel free to suggest another name. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-27 13:00 ` Dmitry Gutov @ 2015-07-27 13:02 ` Dmitry Gutov 2015-07-28 1:21 ` Stephen Leake 1 sibling, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-27 13:02 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/27/2015 04:00 PM, Dmitry Gutov wrote: > Maybe "global" is not the right word, but I'm quite confident that > if I ask an average user what a "search across project" action should ^ replace ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-27 13:00 ` Dmitry Gutov 2015-07-27 13:02 ` Dmitry Gutov @ 2015-07-28 1:21 ` Stephen Leake 2015-07-28 11:05 ` Stephen Leake 2015-07-28 14:18 ` Dmitry Gutov 1 sibling, 2 replies; 87+ messages in thread From: Stephen Leake @ 2015-07-28 1:21 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/26/2015 09:57 PM, Stephen Leake wrote: > >> A list of project names. Depending on the project manager, that implies >> a list of project files in one syntax or another, each of which >> specifies a set of project directories (explicitly or implicitly). > > Could you give an example? Ada, gradle. > Since they're referred only by names, this sounds like a list of > dependencies on the installed packages. Those we usually don't want to > edit, and I'd classify them as system includes. I don't know who this "we" is, but I usually structure a large project as a main with several lower level libraries, all of which I maintain, and I edit them all together to implement new functionality in main. So that has to be a choice. >> Trust what the project file says. The user told Emacs to use the project >> file; it should not try to be smarter. > > So, xref-find-regexp won't call project-root. Got it. Actually, 'xref-find-regexp' should be named 'project-find-regexp' (or prj-find-regexp). Keep 'xref' for strictly cross-reference stuff; 'project' is for more general project stuff. Either that, or drop 'project' and just call it all 'xref'. >> If the user wants documentation in the project, then it is in the >> project file. If the project manager tool is brain-dead and can't handle >> documentation, the the user can switch tools. > > In that case, will the user add the documentation directory to the > search-path section of the project file? Yes, except most backend project files don't have things called "search-path", since they are not for editors; it would be "source-path". > To sum up, I see a strong case for project-search-path as you > described it (maybe under a different name; like project-directories, > as it is now), and so far, no real case for project-root. Ok, we are converging. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 1:21 ` Stephen Leake @ 2015-07-28 11:05 ` Stephen Leake 2015-07-28 14:33 ` Dmitry Gutov 2015-07-28 14:18 ` Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-07-28 11:05 UTC (permalink / raw) To: emacs-devel Stephen Leake <stephen_leake@stephe-leake.org> writes: > Dmitry Gutov <dgutov@yandex.ru> writes: > >> On 07/26/2015 09:57 PM, Stephen Leake wrote: >> >>> A list of project names. Depending on the project manager, that implies >>> a list of project files in one syntax or another, each of which >>> specifies a set of project directories (explicitly or implicitly). >> >> Could you give an example? > > Ada, gradle. > >> Since they're referred only by names, this sounds like a list of >> dependencies on the installed packages. Those we usually don't want to >> edit, and I'd classify them as system includes. > > I don't know who this "we" is, but I usually structure a large project > as a main with several lower level libraries, all of which I maintain, > and I edit them all together to implement new functionality in main. So > that has to be a choice. > >>> Trust what the project file says. The user told Emacs to use the project >>> file; it should not try to be smarter. >> >> So, xref-find-regexp won't call project-root. Got it. > > Actually, 'xref-find-regexp' should be named 'project-find-regexp' (or > prj-find-regexp). Keep 'xref' for strictly cross-reference stuff; > 'project' is for more general project stuff. > > Either that, or drop 'project' and just call it all 'xref'. > >>> If the user wants documentation in the project, then it is in the >>> project file. If the project manager tool is brain-dead and can't handle >>> documentation, the the user can switch tools. >> >> In that case, will the user add the documentation directory to the >> search-path section of the project file? > > Yes, except most backend project files don't have things called > "search-path", since they are not for editors; it would be > "source-path". > >> To sum up, I see a strong case for project-search-path as you >> described it (maybe under a different name; like project-directories, >> as it is now), and so far, no real case for project-root. > > Ok, we are converging. I just realized project.el is in emacs master; I thought it was only in the experimental branch. So some specific change proposals: - Rename 'project-directories' to 'project-root-directories' or 'project-roots'. The current project root should always be first in the list. - 'project-search-path' should not include 'project-root-directories'. - 'elisp-search-path' should not include 'package-user-dir'; package-user-dir is not an elisp source directory, and load-path already includes the subdirs of it, if packages are initialized. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 11:05 ` Stephen Leake @ 2015-07-28 14:33 ` Dmitry Gutov 2015-07-28 15:45 ` Stephen Leake 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-28 14:33 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/28/2015 02:05 PM, Stephen Leake wrote: > I just realized project.el is in emacs master; I thought it was only in > the experimental branch. It would be cleaner to have it separate, but being able to dogfood it now is quite handy. > So some specific change proposals: > > - Rename 'project-directories' to 'project-root-directories' or > 'project-roots'. Either is fine with me, but note that y having "root" in the name we give up on adding a special "project-root" function in the future, one that you described earlier. > The current project root should always be first in the list. > - 'project-search-path' should not include 'project-root-directories'. I say we document it like that (or say that it's "allowed not to include"), but the consumers will still take care of the duplication (call project--prune-directories). That's easier on the implementations. > - 'elisp-search-path' should not include 'package-user-dir'; > package-user-dir is not an elisp source directory, and load-path > already includes the subdirs of it, if packages are initialized. What's the harm? It's an optimization: this way, we only call 'find' once per elpa dir, instead of doing it for each installed package. Not sure what's the overhead on that, but it seems wasteful. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 14:33 ` Dmitry Gutov @ 2015-07-28 15:45 ` Stephen Leake 2015-07-28 16:25 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-07-28 15:45 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/28/2015 02:05 PM, Stephen Leake wrote: > >> - Rename 'project-directories' to 'project-root-directories' or >> 'project-roots'. > > Either is fine with me, but note that y having "root" in the name we > give up on adding a special "project-root" function in the future, one > that you described earlier. No, I'm happy with this: 'project-root'; find the root directory of the current project. 'project-roots'; return a list of related project roots. That's what renaming 'project-directories' to 'project-roots' gives us. Although some will object to the small name difference between 'project-root' and 'project-roots'. >> - 'project-search-path' should not include 'project-root-directories'. > > I say we document it like that (or say that it's "allowed not to > include"), but the consumers will still take care of the duplication > (call project--prune-directories). That's easier on the > implementations. Are you agreeing that the default implementation should not include project-root-directories? The point is that if you only customize by setting project-search-path-function, you can't exclude project-directories. And I thought we agreed that project-search-path is disjoint from project-directories. >> - 'elisp-search-path' should not include 'package-user-dir'; >> package-user-dir is not an elisp source directory, and load-path >> already includes the subdirs of it, if packages are initialized. > > What's the harm? It's an optimization: this way, we only call 'find' > once per elpa dir, instead of doing it for each installed package. Not > sure what's the overhead on that, but it seems wasteful. As with all such optimizations, it can only be justified by actual measurement. But I missed the fact that the default project-search-dirs uses 'project--prune-directories' to throw away the subdirs; so while 'elisp-search-dirs' contains redundant directories, 'project-search-dirs' does not. I find it confusing mixing two customization mechanisms; I started another thread on that. It would be less confusing if the default definition of project-search-path did _not_ call project--prune-directories, but elisp-search-path _did_. Then project--prune-directories should be renamed without the '--'. The doc string for project-search-path says "source directories", not "source root directories". That needs to be fixed. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 15:45 ` Stephen Leake @ 2015-07-28 16:25 ` Dmitry Gutov 2015-07-29 1:36 ` Stephen Leake 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-28 16:25 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/28/2015 06:45 PM, Stephen Leake wrote: > 'project-roots'; return a list of related project roots. > > That's what renaming 'project-directories' to 'project-roots' gives us. > Although some will object to the small name difference between > 'project-root' and 'project-roots'. If project-roots does not include project-root (so we limit the notion of "current project" to just one directory tree), it would be better to rename that to project-related-projects, and return project instances in there. This way, the consumer can inquire about each related project's dependencies, as well as call project-ignores on each of them. But that becomes more complex, because then you'd expect the consumer to call project-related-projects on each related project ask well, and traverse the whole tree, handle loops, etc. I think it'll be better to incorporate all related projects into the notion of the current project, return all project roots on equal rights, give project-ignores an argument (one of the roots), and introduce project-relativize which, like described previously, will return (ROOT . RELATIVE-PATH). In this scheme, project-root is not needed, because we want to treat all roots somewhat equally, and also because project-root shouldn't use default-directory, or buffer-file-name. > Are you agreeing that the default implementation should not include > project-root-directories? Yes. > The point is that if you only customize by setting > project-search-path-function, you can't exclude project-directories. > > And I thought we agreed that project-search-path is disjoint from > project-directories. Like I said, "allowed not to include". Not matter if emacs-lisp-mode sets project-search-path-function, or provides a specialized implementation of project-search-path, it doesn't know the current project roots, so it can't exclude them. > As with all such optimizations, it can only be justified by actual > measurement. It's an easy one, though. > It would be less confusing if the default definition of > project-search-path did _not_ call project--prune-directories, but > elisp-search-path _did_. Some caller up the stack will have to call project--prune-directories anyway, because we don't want to guarantee that search-path and project-roots don't mix (see above). However, we could provide a public function in project.el that will combine both. I having hard time coming up with the name, though: project-full-search-path doesn't sound too good. > Then project--prune-directories should be> renamed without the '--'. I've been waiting for a better name: there are several -prune-directories function in Emacs already, in different packages, all with different semantics. > The doc string for project-search-path says "source directories", not > "source root directories". That needs to be fixed. All right. To me, the two sound equivalent, and I meant it like that. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 16:25 ` Dmitry Gutov @ 2015-07-29 1:36 ` Stephen Leake 2015-07-29 2:10 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-07-29 1:36 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/28/2015 06:45 PM, Stephen Leake wrote: > > I think it'll be better to incorporate all related projects into the > notion of the current project, return all project roots on equal > rights, That is certainly what Ada mode does, and gradle. > give project-ignores an argument (one of the roots), and > introduce project-relativize which, like described previously, will > return (ROOT . RELATIVE-PATH). > > In this scheme, project-root is not needed, because we want to treat > all roots somewhat equally, and also because project-root shouldn't > use default-directory, or buffer-file-name. So you would keep project-roots, but not project-root. I can live with that (Ada mode needs neither :). > Not matter if emacs-lisp-mode > sets project-search-path-function, or provides a specialized > implementation of project-search-path, it doesn't know the current > project roots, so it can't exclude them. How will the elisp backend define project-roots? Since it doesn't know anything about project-roots, it should either throw an error, or return nil. On the other hand, (project-search-path (current-project)) in a .el buffer on my machine returns: ("c:/Projects/emacs/master-build-mingw64/lisp/" "c:/Projects/emacs/master/" "c:/Projects/emacs_stephe.main/emacs_stephe/" "c:/Projects/emacs_stephe.main/emacs_stephe_site_lisp/" "c:/Projects/org.emacs.ada-mode/" "c:/Projects/org.emacs.dvc/lisp/" "c:/home/stephe/.emacs.d/elpa/") Those are all 'project-roots' as far as I can tell; they each represent a logically and physically separate collection of elisp source files (except that master-build-mingw64/lisp/ actually has no elisp source files). So for elisp, it seems we are definining project-search-path to be project-roots. Which goes a long way to explaining why this has been so confusing. On the other hand, "(project-directories (project-current))" returns the single root that contains the current .el file; the same as "(project-root (project-current))". Why does project-directories not return a list? Currently, the only code that uses project-directories is xref-find-regexp, and it concats it with project-search-path, so it confuses the two. Ada mode has no need of project-roots. It seems elisp also has no need of project-roots. JDE (Java Development Environment) has no need of project-roots. >> It would be less confusing if the default definition of >> project-search-path did _not_ call project--prune-directories, but >> elisp-search-path _did_. > > Some caller up the stack will have to call project--prune-directories > anyway, because we don't want to guarantee that search-path and > project-roots don't mix (see above). The top-level generic API should be agnostic about that; only the backend knows precisely what project-search-path and project-roots are. In elisp and Ada, project-roots is simply undefined. Ada mode wants search-path to be flat, _not_ containing roots. So it does _not_ what something to call project--prune-directories. > However, we could provide a > public function in project.el that will combine both. Why? I still have not seen an actual use case for project-roots; it seems the simplest thing is to simply drop it. >> The doc string for project-search-path says "source directories", not >> "source root directories". That needs to be fixed. > > All right. To me, the two sound equivalent, and I meant it like that. The difference is whether recursion into subdirectories is required (in this case, allowed); that is always worth documenting; it is normally an option in search functions (think grep vs grep-find; semantic-symref-find-references-by-name has a :scope argument). -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-29 1:36 ` Stephen Leake @ 2015-07-29 2:10 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-29 2:10 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/29/2015 04:36 AM, Stephen Leake wrote: > So you would keep project-roots, but not project-root. I can live with > that (Ada mode needs neither :). It's fine if some projects define project-roots to nil. Unfortunately, as I imagine it, xref-query-replace won't work for them. Of course, we could add a preference (turned off by default) that xref-query-replace walks the matches in the search-path, too, not just in project-roots. > How will the elisp backend define project-roots? It won't. Creating a project backend for Elisp seems pretty much impossible (or rather useless), from where I'm standing. That's why emacs-lisp-mode sets project-search-path-function instead: the VC project backend uses it automatically. > Since it doesn't know anything about project-roots, it should either > throw an error, or return nil. But that's exactly the problem: if I'm working on a third-party Elisp project, it has its own repository, there's no guarantee that its root directory is added to load-path. If all Elisp code is in the lisp subdirectory (and it's in load-path), if I call xref-find-regexp, it would miss all files on the level above. > On the other hand, (project-search-path (current-project)) in a .el > buffer on my machine returns: > > ("c:/Projects/emacs/master-build-mingw64/lisp/" > "c:/Projects/emacs/master/" > "c:/Projects/emacs_stephe.main/emacs_stephe/" > "c:/Projects/emacs_stephe.main/emacs_stephe_site_lisp/" > "c:/Projects/org.emacs.ada-mode/" > "c:/Projects/org.emacs.dvc/lisp/" > "c:/home/stephe/.emacs.d/elpa/") > > Those are all 'project-roots' as far as I can tell; they each represent > a logically and physically separate collection of elisp source files > (except that master-build-mingw64/lisp/ actually has no elisp source files). No, these are just search-path. We don't know where the roots of most of these "projects" are. > On the other hand, "(project-directories (project-current))" returns the > single root that contains the current .el file; the same as > "(project-root (project-current))". Why does project-directories not > return a list? It returns a one-element list. > Currently, the only code that uses project-directories is > xref-find-regexp, and it concats it with project-search-path, so it > confuses the two. Right. Sorry, that must indeed be confusing. xref-query-replace does not discriminate against project-search-path yet. > Ada mode has no need of project-roots. It seems elisp also has no need > of project-roots. JDE (Java Development Environment) has no need of > project-roots. A common Java project usually has one root: the directory containing its project file (build.xml, pom.xml, or so on). Even so the project implementation could provide a way to open several projects together, if explicitly told so by the user. Then it would return them all in project-roots. >> Some caller up the stack will have to call project--prune-directories >> anyway, because we don't want to guarantee that search-path and >> project-roots don't mix (see above). > > The top-level generic API should be agnostic about that; only the > backend knows precisely what project-search-path and project-roots are. > In elisp and Ada, project-roots is simply undefined. So? The caller would do what xref-find-regexp does now: (project--prune-directories (nconc (project-roots proj) (project-search-path proj))) > Ada mode wants search-path to be flat, _not_ containing roots. So it > does _not_ what something to call project--prune-directories. It wouldn't hurt anyway. >> However, we could provide a >> public function in project.el that will combine both. > > Why? So that the caller doesn't need to call project--prune-directories itself. > I still have not seen an actual use case for project-roots; it seems the > simplest thing is to simply drop it. I have provided a few, along the way of this discussion. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 1:21 ` Stephen Leake 2015-07-28 11:05 ` Stephen Leake @ 2015-07-28 14:18 ` Dmitry Gutov 2015-07-28 16:15 ` Stephen Leake 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-28 14:18 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/28/2015 04:21 AM, Stephen Leake wrote: >> Could you give an example? > > Ada, gradle. I meant an example (or several) of those declarations from an actual project file. Here an excerpt from an imaginary Gemfile: gem 'foo', '~> 1.2.3' gem 'bar', path: '../bar' Having the latter kind of declaration might not be considered the best idea, but it's often nice to be able to do that. Anyway, I'm sure there are corresponding configurations in other ecosystems that are more idiomatic. > I don't know who this "we" is, but I usually structure a large project > as a main with several lower level libraries, all of which I maintain, > and I edit them all together to implement new functionality in main. So > that has to be a choice. IME, aside from a few libraries one (person/team/company) maintains themselves, there's also a long list of external dependencies. In the example above, 'bar' will be one of project-directory-roots, and 'foo' will only be in project-search-path. > Actually, 'xref-find-regexp' should be named 'project-find-regexp' (or > prj-find-regexp). Keep 'xref' for strictly cross-reference stuff; Sounds good to me. I'll make that change as soon as xref has a suitable public interface for that. > 'project' is for more general project stuff. It's also used in the xref-find-references implementations in elisp-mode and etags, which I'm inclined to converge into one default implementation that uses the info from the project package. I think that's fine. > Either that, or drop 'project' and just call it all 'xref'. project is foremost intended to be used for third-party code. It's a feature that's been requested for a while. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 14:18 ` Dmitry Gutov @ 2015-07-28 16:15 ` Stephen Leake 2015-07-28 18:44 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-07-28 16:15 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/28/2015 04:21 AM, Stephen Leake wrote: > >>> Could you give an example? >> >> Ada, gradle. > I meant an example (or several) of those declarations from an actual > project file. From an Ada project file: aggregate project Books_Agg is for Project_Path use ("../../../org.stephe_leake.sal/build/release", "../../../org.stephe_leake.gtk/build/release", "../../../org.stephe_leake.makerules"); for Project_Files use ("books.gpr"); end Books_Agg; with "gtkada"; with "gnatcoll_sqlite"; with "sal"; with "sal_gtk"; with "standard_common"; project Books is for Source_Dirs use ("../../source", "../../test"); ... end Books; Project_Path says where to find projects (there is also an implicit system location); 'with "...";' says what projects to include. In this case, "gtkada" and "gnatcoll_sqlite" are in the system location, and are read only; sal, sal_gtk and standard_common read/write. But the Ada project tool (as provided by the compiler vendor) makes no distinction between "system" and "user" libraries; more precisely, it relies on the file system read/write permissions to do that. From an Android gradle file: dependencies { compile project(':licensesdialoglibrary') compile project(':quickScroll') compile project(':velocityviewpagerlibrary') compile project(':dragsortlistviewlibrary') compile project(':viewpagerindicatorlibrary') compile project(':circularImageView') compile project(':picasso') compile files('libs/android-async-http-1.4.2-66-g4b6eb97.jar') compile files('libs/commons-io-2.4.jar') compile files('libs/commons-lang3-3.1.jar') compile files('libs/commons-logging.jar') compile files('libs/dashclock-api-r2.0.jar') compile files('libs/google-http-client-1.16.0-rc.jar') compile files('libs/google-http-client-android-1.16.0-rc.jar') compile files('libs/jaudiotagger-2.0.4-20111207.115108-15.jar') compile files('libs/libGoogleAnalyticsServices.jar') compile files('libs/universal-image-loader-1.9.3-with-sources.jar') compile files('libs/com.haarman.listviewanimations-2.6.0.jar') compile files('libs/nineoldandroids-2.4.0.jar') compile files('libs/renderscript-v8.jar') compile 'com.android.support:support-v4:+' compile 'com.google.android.gms:play-services:+' } Some of these are read/write, some are read-only; gradle does not have a mechanism to indicate that. > Here an excerpt from an imaginary Gemfile: > > gem 'foo', '~> 1.2.3' > gem 'bar', path: '../bar' > > Having the latter kind of declaration might not be considered the best > idea, but it's often nice to be able to do that. Anyway, I'm sure > there are corresponding configurations in other ecosystems that are > more idiomatic. The point is that the abstract project API must handle all of these. In the case of Ada, there is a command-line tool that parses the project file and provides the info that Emacs needs. >> I don't know who this "we" is, but I usually structure a large project >> as a main with several lower level libraries, all of which I maintain, >> and I edit them all together to implement new functionality in main. So >> that has to be a choice. > > IME, aside from a few libraries one (person/team/company) maintains > themselves, there's also a long list of external dependencies. Yes, we are agreeing; some dependencies are read/write, some are read-only. How many of each, and whether they are "system" or not, are minor issues. > In the example above, 'bar' will be one of project-directory-roots, > and 'foo' will only be in project-search-path. Why? What effect does that have on what you can do with 'foo' vs 'bar'? How can you tell that just from the gem file? I'm guessing you are saying "foo is maintained by the same team as the main project (ie read/write), bar is maintained elsewhere (ie read-only)". So then files in project-directories are read-only, files in project-search-path are read/write. But the current default implementation of project-search-path _inludes_ project-directories, so this is inconsistent. And we've said that project-directories includes the main project root, so that's also inconsistent. So I don't understand what you are doing here. >> Actually, 'xref-find-regexp' should be named 'project-find-regexp' (or >> prj-find-regexp). Keep 'xref' for strictly cross-reference stuff; > > Sounds good to me. I'll make that change as soon as xref has a > suitable public interface for that. I don't understand; this requires deleting things from xref, not adding to it. The current implementation of xref-find-regexp uses project-current, xref-collect-matches, and xref--show-xrefs. If you make the implementation of project-find-regexp use xref facilities, then I understand your comment. I was hoping for more separation. I suggest renaming 'xref--show-xrefs' to 'project-show-locations', and have it take a list of locations, not implicitly call xref-find-function. Also move all of the location stuff to project; it's more general than xref. xref-collect-matches uses grep; it can simply be moved to project-collect-matches, along with the xref-*grep functions it uses. The doc string needs to mention subdirs. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 16:15 ` Stephen Leake @ 2015-07-28 18:44 ` Dmitry Gutov 2015-07-29 2:27 ` Stephen Leake 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-28 18:44 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/28/2015 07:15 PM, Stephen Leake wrote: > From an Ada project file: Thank you. > Project_Path says where to find projects (there is also an implicit > system location); 'with "...";' says what projects to include. In this > case, "gtkada" and "gnatcoll_sqlite" are in the system location, and are > read only; sal, sal_gtk and standard_common read/write. But the Ada > project tool (as provided by the compiler vendor) makes no distinction > between "system" and "user" libraries; more precisely, it relies on the > file system read/write permissions to do that. That's unfortunate. But you do specify "custom" locations for sal, sal_gtk and standard_common in the beginning, right? Then a smart enough tool could learn that information. In the meantime, I guess, the Ada projects will have to classify all dependencies together as search-path. > > From an Android gradle file: > > dependencies { > > compile project(':licensesdialoglibrary') > compile files('libs/android-async-http-1.4.2-66-g4b6eb97.jar') > compile 'com.google.android.gms:play-services:+' > } > > Some of these are read/write, some are read-only; gradle does not have a > mechanism to indicate that. jars are obviously build artefacts. Doesn't the difference between compile '...' and compile project('...') signify something, though? >> Here an excerpt from an imaginary Gemfile: >> >> gem 'foo', '~> 1.2.3' >> gem 'bar', path: '../bar' >> > The point is that the abstract project API must handle all of these. In > the case of Ada, there is a command-line tool that parses the project > file and provides the info that Emacs needs. I'm assuming that the build tools in other ecosystems can likewise separate dependencies into search-path and full-on projects. Or, at least, that most of them can, and this separation is meaningful. And that there's no separation along different lines that's more useful than that. > Yes, we are agreeing; some dependencies are read/write, some are read-only. > > How many of each, and whether they are "system" or not, are minor > issues. Actually, neither "system", nor "read/write" terminology strikes me as optimal. "search path" vs. "projects" sounds better: one could edit a file inside the system search path, with purpose, and we're should allow that operation, but not optimize the interface for it. >> In the example above, 'bar' will be one of project-directory-roots, >> and 'foo' will only be in project-search-path. > > Why? What effect does that have on what you can do with 'foo' vs 'bar'? > > How can you tell that just from the gem file? path: '/..' says that 'bar' is a project residing in a nearby directory, and it's probably version-controlled. The fact that I have a Git checkout of 'bar' is a strong indicator that I'm maintaining that project (though not a guarantee, of course). It makes sense to perform read/write operations on version-controlled directories. 'foo', on the other hand, resides in ~/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/foo-1.2.3/, and any edits I make to it are hard to track or push to the development repo. > I'm guessing you are saying "foo is maintained by the same team as the > main project (ie read/write), bar is maintained elsewhere (ie > read-only)". The opposite of that. > So then files in project-directories are read-only, files in > project-search-path are read/write. Ditto. > But the current default implementation of project-search-path _inludes_ > project-directories, so this is inconsistent. Is allowed to include. > And we've said that project-directories includes the main project root, > so that's also inconsistent. Why? > So I don't understand what you are doing here. read/write operations will use project-roots; read-only operations will use project-roots combined with project-search-path. I though we've already settled on that. > I don't understand; this requires deleting things from xref, not adding > to it. xref-find-regexp uses things from xref, such as the output buffer, its major mode, etc. They're not exposed publicly in a proper fashion, yet. > The current implementation of xref-find-regexp uses project-current, > xref-collect-matches, and xref--show-xrefs. Right. So it uses both xref.el and project.el. > If you make the implementation of project-find-regexp use xref > facilities, then I understand your comment. Exactly. > I was hoping for more separation. What kind? > I suggest renaming 'xref--show-xrefs' to 'project-show-locations', and Why project-? The procedure of displaying the locations doesn't have much to do with the project API. The xref API, however, is very much related. > have it take a list of locations, not implicitly call > xref-find-function. The difficulty with that had been the necessity to track the buffers that xref-find-function creates, in order to be able to delete each of them that haven't been used otherwise, in xref-quit. The parallel thread about hidden/temporary buffers should result in solving that in a different way, and then I'll do just that. > xref-collect-matches uses grep; it can simply be moved to > project-collect-matches, along with the xref-*grep functions it uses. > The doc string needs to mention subdirs. I think there's value in providing function, in xref, that returns a list of matches for a given regexp in a given directory, as xref-items. It's not, by itself, project-related, and other xref backends might use it for cheap xref-find-references implementations. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-28 18:44 ` Dmitry Gutov @ 2015-07-29 2:27 ` Stephen Leake 2015-07-29 22:51 ` Dmitry Gutov 2015-07-29 23:11 ` xref display and multiple locations, " Dmitry Gutov 0 siblings, 2 replies; 87+ messages in thread From: Stephen Leake @ 2015-07-29 2:27 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/28/2015 07:15 PM, Stephen Leake wrote: > >> From an Ada project file: > > Thank you. > >> Project_Path says where to find projects (there is also an implicit >> system location); 'with "...";' says what projects to include. In this >> case, "gtkada" and "gnatcoll_sqlite" are in the system location, and are >> read only; sal, sal_gtk and standard_common read/write. But the Ada >> project tool (as provided by the compiler vendor) makes no distinction >> between "system" and "user" libraries; more precisely, it relies on the >> file system read/write permissions to do that. > > That's unfortunate. Why? it is what it is; it works. > But you do specify "custom" locations for sal, > sal_gtk and standard_common in the beginning, right? Then a smart > enough tool could learn that information. I don't know what you mean by "learn"; the user writes the project file, the tools read it. Emacs uses one tool to read the project file, and extract a flat project-search-path (the concatenation of all the Source_Path attributes). > In the meantime, I guess, the Ada projects will have to classify all > dependencies together as search-path. Yes, that's what I've been saying; it's a valid use case. That is also what elisp is doing now. >> From an Android gradle file: >> >> dependencies { >> >> compile project(':licensesdialoglibrary') >> compile files('libs/android-async-http-1.4.2-66-g4b6eb97.jar') >> compile 'com.google.android.gms:play-services:+' >> } >> >> Some of these are read/write, some are read-only; gradle does not have a >> mechanism to indicate that. > > jars are obviously build artefacts. Doesn't the difference between > compile '...' and compile project('...') signify something, though? Yes, but I'm not sure what. I haven't used this enough yet. >>> Here an excerpt from an imaginary Gemfile: >>> >>> gem 'foo', '~> 1.2.3' >>> gem 'bar', path: '../bar' >>> >> The point is that the abstract project API must handle all of these. In >> the case of Ada, there is a command-line tool that parses the project >> file and provides the info that Emacs needs. > > I'm assuming that the build tools in other ecosystems can likewise > separate dependencies into search-path and full-on projects. Or, at > least, that most of them can, and this separation is meaningful. I know of no tool that does that; I do _not_ make that assumption. Where did you get the idea it was meaningful? >>> In the example above, 'bar' will be one of project-directory-roots, >>> and 'foo' will only be in project-search-path. >> >> Why? What effect does that have on what you can do with 'foo' vs 'bar'? >> >> How can you tell that just from the gem file? > > path: '/..' says that 'bar' is a project residing in a nearby > directory, Ok > and it's probably version-controlled. The fact that I have > a Git checkout of 'bar' is a strong indicator that I'm maintaining > that project (though not a guarantee, of course). That's extra information, _not_ in the gem file. > 'foo', on the other hand, resides in > ~/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/foo-1.2.3/, I guess that's what ">.1.2.3" means? It's some ruby-defined library directory in your home directory. Similar to a "system library"? > and any edits I make to it are hard to track or push to the > development repo. Ok, since it's like a system library. >> I'm guessing you are saying "foo is maintained by the same team as the >> main project (ie read/write), bar is maintained elsewhere (ie >> read-only)". > > The opposite of that. Ok. >> So then files in project-directories are read-only, files in >> project-search-path are read/write. > > Ditto. Which is _not_ what the current code does; xref-query-replace (an edit function) operates on the result of xref-find-regexp (among other things), which uses (nconc project-search-path project-directories). >> But the current default implementation of project-search-path _inludes_ >> project-directories, so this is inconsistent. > > Is allowed to include. I'm not talking about the doc string; I'm talking about the actual code, which is what elisp uses. But you've agreed to change that. > read/write operations will use project-roots; read-only operations > will use project-roots combined with project-search-path. I though > we've already settled on that. News to me. If that is the intended use of these two functions, then they should be named project-path-read-write and project-path-read-only; that will be _much_ clearer. It can still be up to the backend to decide whether they are flat or require additional recursion. I've never found the need for that distinction in Ada mode. >> I suggest renaming 'xref--show-xrefs' to 'project-show-locations', and > > Why project-? The procedure of displaying the locations doesn't have > much to do with the project API. The xref API, however, is very much > related. Locations are tied other things besides cross-reference; cross-reference is just one way of producing a list of locations. find-regexp is another way. compilation errors is another way. All of those could use a common location API. 'project' is one place to put it; it could be in its own 'location' namespace. Currently, 'grep-find' and 'compilation' share the same location API. Which is why I suggested that xref also share it. But I do like the current xref location display, so maybe there is a case for them to remain distinct. They serve different purposes; the xref display makes it easy to choose one location from a small list; the compilation display makes it easier to step through all of them in sequence, and it works even for very long lists. There should at least there should be this choice of display; the location data structure could probably be shared (the compilation location structure contains more info). project-find-regexp should use the same location display API as grep-find, since they serve very similar purposes; the user will want to step thru all of them. Or maybe not, in some particular use case. (grep-find could eventually be made obsolete in favor of project-find-regexp). >> have it take a list of locations, not implicitly call >> xref-find-function. > > The difficulty with that had been the necessity to track the buffers > that xref-find-function creates, in order to be able to delete each of > them that haven't been used otherwise, in xref-quit. > > The parallel thread about hidden/temporary buffers should result in > solving that in a different way, and then I'll do just that. Good. >> xref-collect-matches uses grep; it can simply be moved to >> project-collect-matches, along with the xref-*grep functions it uses. >> The doc string needs to mention subdirs. > I think there's value in providing function, in xref, that returns a > list of matches for a given regexp in a given directory, as > xref-items. That's just reimplementing Emacs grep to use the alternate location API; not worth it. > It's not, by itself, project-related, Nor is it xref-related. > and other xref backends might use it for cheap xref-find-references > implementations. After adding path and/or directory recursion support, for searching project-search-path. I hope not; better to say "xref not implemented; use Emacs grep". That will motivate people to implement a real xref backend, as well as immediately explaining why the xref is not accurate. It would be nice if <menu-bar> <tools> <grep> defaulted to the identifier at point. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-29 2:27 ` Stephen Leake @ 2015-07-29 22:51 ` Dmitry Gutov 2015-07-30 8:17 ` Stephen Leake 2015-07-29 23:11 ` xref display and multiple locations, " Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-29 22:51 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/29/2015 05:27 AM, Stephen Leake wrote: > Why? it is what it is; it works. It gives us less information than it could, about the project configuration. >> But you do specify "custom" locations for sal, >> sal_gtk and standard_common in the beginning, right? Then a smart >> enough tool could learn that information. > > I don't know what you mean by "learn"; the user writes the project file, > the tools read it. Figure out somehow, based on the project file and the contents of the file system, hopefully without asking the user more questions. For instance, even if the project file gives no direct indication of which search-path elements are "proper" projects, some Elisp code could look inside each of those directories (and one level above, maybe) and see if there are project files in any of them. And those where there are, promote to project-roots as well. That might be harder for Ada, but should be totally doable for Java-based projects (if build.gradle doesn't provide that info). > Yes, that's what I've been saying; it's a valid use case. > > That is also what elisp is doing now. Elisp is also relying on runtime information. It's not reliable enough, IMO, to use as the basis of xref-query-replace, for instance. Like I mentioned in the other email, the project I've opened right now isn't necessarily in load-path. AdaCore and Gradle are in better positions than that. >> jars are obviously build artefacts. Doesn't the difference between >> compile '...' and compile project('...') signify something, though? > > Yes, but I'm not sure what. I haven't used this enough yet. That might be worth investigating. >> I'm assuming that the build tools in other ecosystems can likewise >> separate dependencies into search-path and full-on projects. Or, at >> least, that most of them can, and this separation is meaningful. > > I know of no tool that does that; I do _not_ make that assumption. Where > did you get the idea it was meaningful? The IDEs usually separate the "current project" (one root with its subtree) from the system dependencies. Especially in the Java world, where there's just too much code to look at. Multi-project builds also exist, but they seem to be usually constrained to the "several projects inside one parent directory" configuration, see "7.3. Multi-project Java build" in https://docs.gradle.org/current/userguide/tutorial_java_projects.html. Also, here's a screenshot of a "project drawer" in a popular IDE: http://blog.pivotal.io/wp-content/uploads/2012/02/intellij_modules.png It showcases a separation of the current project's "modules" from "external libraries". >> and it's probably version-controlled. The fact that I have >> a Git checkout of 'bar' is a strong indicator that I'm maintaining >> that project (though not a guarantee, of course). > > That's extra information, _not_ in the gem file. It's a good assumption. Even if it's not version controlled, it's most likely a working directory anyway, and the developer's just being ignorant about version control. >> 'foo', on the other hand, resides in >> ~/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/foo-1.2.3/, > > I guess that's what ">.1.2.3" means? It's some ruby-defined library > directory in your home directory. Similar to a "system library"? Exactly like a system library. > Which is _not_ what the current code does; xref-query-replace (an edit > function) operates on the result of xref-find-regexp (among other > things), which uses (nconc project-search-path project-directories). Indeed. But that shouldn't be surprising: we are discussing a change proposal, and xref-find-regexp, xref-query-replace, and other related code you can see in master, had all been written before the current iteration of this thread. >>> But the current default implementation of project-search-path _inludes_ >>> project-directories, so this is inconsistent. >> >> Is allowed to include. > > I'm not talking about the doc string; I'm talking about the actual code, > which is what elisp uses. But you've agreed to change that. Ah, yes, of course. I intend to change that. But consider this: elisp-search-path is not ideal (the current project could be not in load-path). So if project-search-path only consults project-search-path-function, by default, what will the default implementation of xref-find-references use? Just (project-search-path (project-current)), or (project-prune-directories (append (project-roots proj) (project-search-path proj))) ? The latter moves the same logic into the caller. The former will give the user false negatives, in the absence of a smart, language-aware project implementation. > If that is the intended use of these two functions, then they should be > named project-path-read-write and project-path-read-only; that will be > _much_ clearer. It can still be up to the backend to decide whether they > are flat or require additional recursion. Err, that sounds silly. Why would you want not to recurse? The current xref-collect-matches implementation unconditionally recurses, and it doesn't seem particularly useful to create a variant that doesn't. > Locations are tied other things besides cross-reference; cross-reference > is just one way of producing a list of locations. > > find-regexp is another way. > > compilation errors is another way. > > All of those could use a common location API. 'project' is one place to > put it; it could be in its own 'location' namespace. I suppose you're right. But until we have a unified location API (which I wouldn't count on appearing anytime soon), xref is the better place. xref-location lives there, and it's the best purpose-agnostic location class that we have now. > Currently, 'grep-find' and 'compilation' share the same location API. > Which is why I suggested that xref also share it. But I do like the > current xref location display, so maybe there is a case for them to > remain distinct. But there's not much API there, is there? To obtain the list of matches from the Grep or Compilation output, one has to re-search-forward in that buffer. >> I think there's value in providing function, in xref, that returns a >> list of matches for a given regexp in a given directory, as >> xref-items. > > That's just reimplementing Emacs grep to use the alternate location API; not > worth it. I think it's useful enough. >> It's not, by itself, project-related, > > Nor is it xref-related. It's location-related, so to speak. Which live in xref, for now. > After adding path and/or directory recursion support, for searching > project-search-path. Not sure what you mean here. > I hope not; better to say "xref not implemented; use Emacs grep". That will > motivate people to implement a real xref backend, as well as immediately > explaining why the xref is not accurate. Why would using M-x grep be better? I think it's better to provide an imperfect implementation that showcases the available features (bindings, commands, presentation). That will better motivate third-party developers to improve on other aspects, such as precision. That's how most open-source projects get their contributors, anyway: when a feature works almost, but not exactly, right. And one shouldn't discount Grep-based searched out of hand. For instance, a simple xref-find-definitions implementations that searches a set of directories for "def funcname", would be a lot better that the absolute nothing we provide out of the box to users who don't use etags. > It would be nice if <menu-bar> <tools> <grep> defaulted to the > identifier at point. M-x report-emacs-bug? That seems unrelated. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-29 22:51 ` Dmitry Gutov @ 2015-07-30 8:17 ` Stephen Leake 2015-07-31 0:15 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-07-30 8:17 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/29/2015 05:27 AM, Stephen Leake wrote: > >>> But you do specify "custom" locations for sal, >>> sal_gtk and standard_common in the beginning, right? Then a smart >>> enough tool could learn that information. >> >> I don't know what you mean by "learn"; the user writes the project file, >> the tools read it. > > Figure out somehow, based on the project file and the contents of the > file system, hopefully without asking the user more questions. No need to "figure it out"; the search path for project files is right there in the file: for Project_Path use ("../../../org.stephe_leake.sal/build/release", "../../../org.stephe_leake.gtk/build/release", "../../../org.stephe_leake.makerules"); That's where the project files for sal, sal_gtk, standard_common are found. This is interpreted relative to the directory the project file is found in. > For instance, even if the project file gives no direct indication of > which search-path elements are "proper" projects, some Elisp code > could look inside each of those directories (and one level above, > maybe) and see if there are project files in any of them. And those > where there are, promote to project-roots as well. > That might be harder for Ada, but should be totally doable for > Java-based projects (if build.gradle doesn't provide that info). For Ada project files, everything is crystal clear; anything mentioned in "with" is a proper project; nothing else can be. No searching involved. Same for gradle; every build.gradle file defines a proper project; nothing else does. > But consider this: elisp-search-path is not ideal (the current project > could be not in load-path). I don't understand; the user sets load-path. If the user wants to use project-* and xref-* functions with the current project, why is it not in load-path? That's just bad user configuration; no amount of elisp intelligence can fix that. xref-find-definitions for elisp relies on the code being loaded, so using it without the code being loaded makes no sense. >> It can still be up to the backend to decide whether they >> are flat or require additional recursion. > > Err, that sounds silly. Why would you want not to recurse? 1) Because many Emacs functions don't recurse. For example, I'd like to add: (defun project-find-file () "Prompt for a file name, find it on project-search-path." (interactive) (let ((filename (completing-read "file: " (apply-partially 'locate-file-completion-table (project-search-path (project-current)) nil)))) (find-file (locate-file filename (project-search-path (project-current)) nil)))) This works fine with "load-path" instead of "project-search-path". But it won't work unless project-search-path is flat; locate-file-completion-table and locate-file do not recurse. 2) As I've mentioned before, not recursing allows computing the set of included/excluded directories once, instead of computing it again on each recursion. 3) Because some tools provide and expect a flat search path (AdaCore tools, gcc include path, emacs load-path, compilation-search-path, ff-search-directories). It is not at all clear how to convert a flat path into a recursive path with ignores; the solution you have for elisp load-path is a kludge, and could easily be wrong for some users. > The current xref-collect-matches implementation unconditionally > recurses, and it doesn't seem particularly useful to create a variant > that doesn't. The backends I'm aware of require flat paths (no recursion). So unless some plausible backend actually requires recursion, it makes more sense to assume recursion is not required. >> Locations are tied other things besides cross-reference; cross-reference >> is just one way of producing a list of locations. >> >> find-regexp is another way. >> >> compilation errors is another way. >> >> All of those could use a common location API. 'project' is one place to >> put it; it could be in its own 'location' namespace. > > I suppose you're right. But until we have a unified location API > (which I wouldn't count on appearing anytime soon), xref is the better > place. > > xref-location lives there, and it's the best purpose-agnostic location > class that we have now. So move it to "location" instead of "xref". The longest journey starts with one step ... -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-30 8:17 ` Stephen Leake @ 2015-07-31 0:15 ` Dmitry Gutov 2015-07-31 16:13 ` Stephen Leake 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-07-31 0:15 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/30/2015 11:17 AM, Stephen Leake wrote: > For Ada project files, everything is crystal clear; anything mentioned > in "with" is a proper project; nothing else can be. No searching > involved. Fantastic, then Ada gives you all you need to implement both project-search-path and project-roots. > I don't understand; the user sets load-path. If the user wants to use > project-* and xref-* functions with the current project, why is it not > in load-path? It's not always the current practice, AFAIK. And adding it to load-path interactively is not something we should force the users to do. Consider that when xref-find-references doesn't find anything, it will say "nothing found", without further explanations, leaving the user to puzzle out why and what. We should err on the side of showing too much, rather than too little. > xref-find-definitions for elisp relies on the code being loaded, so > using it without the code being loaded makes no sense. One can also load the code without adding the file to load-path. For instance, via eval-buffer. > 1) Because many Emacs functions don't recurse. For example, I'd like to add: > > (defun project-find-file () > "Prompt for a file name, find it on project-search-path." > (interactive) > (let ((filename > (completing-read > "file: " > (apply-partially > 'locate-file-completion-table > (project-search-path (project-current)) > nil)))) > > (find-file (locate-file filename (project-search-path (project-current)) nil)))) > > This works fine with "load-path" instead of "project-search-path". But > it won't work unless project-search-path is flat; > locate-file-completion-table and locate-file do not recurse. I guess you'd need to implement a "flattening" function for that, indeed. I don't really see myself using this command, though. > 2) As I've mentioned before, not recursing allows computing the set of > included/excluded directories once, instead of computing it again on > each recursion. And I countered that, for many commands, computing that set is unlikely to be the bottleneck. And cache invalidation is not user-friendly. > 3) Because some tools provide and expect a flat search path (AdaCore > tools, gcc include path, emacs load-path, compilation-search-path, > ff-search-directories). That's not really true. There are many directories inside /usr/include, but C_INCLUDE_PATH won't mention most of them. Instead, #include <postgresql/pg_config> will use /usr/include/postgresql/pg_config.h, as long as /usr/include is in C_INCLUDE_PATH. There's only one directory in Emacs load-path with "cedet" in its name, but (require 'semantic/db) will load lisp/cedet/semantic/db.el. You're right about compilation-search-path and ff-search-directories, though. It seems to be true for the Ada source search path as well, but since /usr/lib/gcc/x86_64-linux-gnu/4.9/adainclude/ has no subdirectories, the point is moot. Traversing it recursively wouldn't make anything worse. > It is not at all clear how to convert a flat path into a recursive path > with ignores; the solution you have for elisp load-path is a kludge, > and could easily be wrong for some users. I'd like to see one of those users first. >> xref-location lives there, and it's the best purpose-agnostic location >> class that we have now. > > So move it to "location" instead of "xref". The longest journey starts > with one step ... Until it's used in at least two different places, creating a separate package for it is, I think, premature. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-31 0:15 ` Dmitry Gutov @ 2015-07-31 16:13 ` Stephen Leake 2015-08-01 0:57 ` Dmitry Gutov 2015-08-01 1:14 ` Per-language project-search-path, was: " Dmitry Gutov 0 siblings, 2 replies; 87+ messages in thread From: Stephen Leake @ 2015-07-31 16:13 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/30/2015 11:17 AM, Stephen Leake wrote: > >> I don't understand; the user sets load-path. If the user wants to use >> project-* and xref-* functions with the current project, why is it not >> in load-path? > > It's not always the current practice, AFAIK. I still don't understand why. But I've proposed `project-add-search-path' (below and in another email), so that can be used to handle this case. > And adding it to load-path interactively is not something we should > force the users to do. Yes, if the user doesn't want a file in load-path, they don't put it there. But then they will have to use project-add-search-path instead. We have to expect some help from the user to indicate what's in the project! > Consider that when xref-find-references doesn't find anything, it will > say "nothing found", without further explanations, leaving the user to > puzzle out why and what. We should err on the side of showing too > much, rather than too little. Sorry, I can't guess what you have in mind that the user is doing here. What is the user searching for, what did they do to specify project-search-path, and what do they expect xref-find-references to show? >> 1) Because many Emacs functions don't recurse. For example, I'd like to add: >> >> (defun project-find-file () >> "Prompt for a file name, find it on project-search-path." >> (interactive) >> (let ((filename >> (completing-read >> "file: " >> (apply-partially >> 'locate-file-completion-table >> (project-search-path (project-current)) >> nil)))) >> >> (find-file (locate-file filename (project-search-path (project-current)) nil)))) >> >> This works fine with "load-path" instead of "project-search-path". But >> it won't work unless project-search-path is flat; >> locate-file-completion-table and locate-file do not recurse. > > I guess you'd need to implement a "flattening" function for that, > indeed. If project-search-path is recursive, yes. Which is certainly possible to implement here; I propose a "recursive" project flag below. If that's set, this function can use directory-files and directories in project-ignores to do the recursion. The predicate for locate-file-completion-table (set to nil above) needs to handle the files in project-ignores as well. > I don't really see myself using this command, though. That doesn't mean it's not a valid use case; you need to consider all users. I often want to look at project.el (or xref.el, ada-mode.el, etc); I bind this function (currently using load-path) to M-F12, so M-F12 proj <ret> finds it for me easily. The Ada user group requested this function for Ada projects (it has a similar implementation). >> 2) As I've mentioned before, not recursing allows computing the set of >> included/excluded directories once, instead of computing it again on >> each recursion. > > And I countered that, for many commands, computing that set is > unlikely to be the bottleneck. And cache invalidation is not > user-friendly. Which is faster/friendlier depends on the details, so we'll have to wait for actual implementations. All I'm asking is that we do not make design decisions at the generic project API level that rule out this approach. >> 3) Because some tools provide and expect a flat search path (AdaCore >> tools, gcc include path, emacs load-path, compilation-search-path, >> ff-search-directories). > > That's not really true. > > There are many directories inside /usr/include, but C_INCLUDE_PATH > won't mention most of them. Instead, #include <postgresql/pg_config> > will use /usr/include/postgresql/pg_config.h, as long as /usr/include > is in C_INCLUDE_PATH. Yes. So if the user wants to search all relevant includes for some declaration, they have to treat C_INCLUDE_PATH as recursive. But the compiler (which _defines_ C_INCLUDE_PATH) does _not_ treat it as recursive. So it's confused. > There's only one directory in Emacs load-path with "cedet" in its > name, but (require 'semantic/db) will load lisp/cedet/semantic/db.el. Similar situation: `load' does not treat `load-path' as recursive, but text searches need to. I had not considered that. I think cedet is the only exception to the 'flat load-path' rule (at least in core emacs); I hope it goes away, and is not copied by other projects. I would provide a special syntax to handle this in project-add-search-path: (defun project-add-search-path (project path &optional make-recursive) "Add PATH (a list of directories) to the search path for PROJECT. If a directory in PATH ends in \"/**\", also add all directories under that directory. If MAKE-RECURSIVE is non-nil, the full project search path is pruned so that only highest-level directories are present, and all uses of `project-search-path' recurse into subdirectories. In this case, it is likely you will also need to specify ignored directories with `project-add-ignores'. If MAKE-RECURSIVE is nil, uses of `project-search-path' no not recurse into subdirectories." ... ) The Ada project files support the ** sytnax. User projects can have directory structures for which recursive search from the vc root is wrong: myproj/ .git/ .gitignore build/ Makefile ada_project.gpr *.exe obj/ # contains object files for ada files lisp/ # elisp files for emacs-25+ lisp-emacs-24/ # elisp files defining emacs-25 functions lisp-emacs-23/ ada/ # ada files The elisp project (depending on current Emacs version) will only want to search one of: (lisp) (lisp lisp-emacs-24) (lisp lisp-emacs-23) The Ada project will only want to search the "ada" and "build" directories (for the Makefile and Ada project file). There will be cases where the user will want to search all files; then a separate project that uses project-add-search-path to add all dirs would be appropriate. No search should include the "obj" directory. This could be handled if the user can explicitly specify project ignores (not just rely on .gitignore etc; git does _not_ ignore the lisp-emacs-* and ada directories). More complicated structures are harder to handle with ignores; you'll end up with a long flat list of ignores, so it might be simpler to have a flat list of includes instead. > You're right about compilation-search-path and ff-search-directories, > though. > > It seems to be true for the Ada source search path as well, but since > /usr/lib/gcc/x86_64-linux-gnu/4.9/adainclude/ has no subdirectories, > the point is moot. Traversing it recursively wouldn't make anything > worse. User Ada projects can easily have directory structures for which recursive search from the vc root would be wrong; see above. It would be surprising to an Ada user (used to flat search paths), if the Emacs project silently started using recursive search paths. >> It is not at all clear how to convert a flat path into a recursive path >> with ignores; the solution you have for elisp load-path is a kludge, >> and could easily be wrong for some users. > > I'd like to see one of those users first. Me, and many other teams at NASA - structures similar to the above, where the non-included directories can also be release version, debug/release options, target hardware/os option, compiler vendor/version, etc. The jdee project has lisp and java directories. The kludge is in the root implementation of project-search-path, so it applies to all project backends that only customize project-search-path-function. I have no objection to allowing some project backends to use recursive path/ignore dirs; I just want to also allow some backends to use non-recursive path/ignore files. Or the user can choose, via project-add-search-path above. >>> xref-location lives there, and it's the best purpose-agnostic location >>> class that we have now. >> >> So move it to "location" instead of "xref". The longest journey starts >> with one step ... > > Until it's used in at least two different places, creating a separate > package for it is, I think, premature. Right. We now have those two places; xref-find-definitions, project-find-regexp. Unless project-find-regexp uses the compile location data structure, which would probably be better. Although it would be even better if project-find-regexp uses the compile location data structure wrapped in a location-compile class, derived from the root location class, renamed from xref-location. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-07-31 16:13 ` Stephen Leake @ 2015-08-01 0:57 ` Dmitry Gutov 2015-08-01 9:50 ` Stephen Leake 2015-08-01 1:14 ` Per-language project-search-path, was: " Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-08-01 0:57 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/31/2015 07:13 PM, Stephen Leake wrote: > I often want to look at project.el (or xref.el, ada-mode.el, etc); I > bind this function (currently using load-path) to M-F12, so M-F12 > proj <ret> finds it for me easily. To reach the same result, I currently either use ido-switch-buffer (with ido-use-virtual-buffers set to t) or projectile-find-file (which asks the user to choose between files in the project, based on file names relative to the root). The key thing here is substring matching: I also simply type "project." and see lisp/progmodes/project.el as the only completion. Using relative paths also helps with navigating to files with the same name inside different directories. > Which is faster/friendlier depends on the details, so we'll have to wait > for actual implementations. All I'm asking is that we do not make design > decisions at the generic project API level that rule out this approach. We do have to make some decisions, though. For instance, supporting both recursive and non-recursive elements in search-path will require a separate code path for each function that uses it. As such, it will require a separate code path in xref-collect-matches and an alternative for xref--rgrep-command. >> There are many directories inside /usr/include, but C_INCLUDE_PATH >> won't mention most of them. Instead, #include <postgresql/pg_config> >> will use /usr/include/postgresql/pg_config.h, as long as /usr/include >> is in C_INCLUDE_PATH. > > Yes. So if the user wants to search all relevant includes for some > declaration, they have to treat C_INCLUDE_PATH as recursive. But the > compiler (which _defines_ C_INCLUDE_PATH) does _not_ treat it as > recursive. So it's confused. We need to search files in subdirectories as well, and that is the key distinction. The fact that compiler treats the roots in a special way, however, can be important. Because of that, the list of roots contains information that can be lost after a naive translation to a list of "flat" directories. For instance, if you have the roots, you can search them linearly and tell whether there's a file corresponding to 'postgresql/pg_config' inside one of them. With a flat list, we could only give an imprecise answer to that question, and the search might have to perform extra work. > I think cedet is the only exception to the 'flat load-path' rule (at > least in core emacs); I hope it goes away, and is not copied by other > projects. Even though it's not very popular, it's unlikely to go away. But in any case, the corresponding path variables in all other ecosystems that I know of (Ruby, Python, Java, Node.js) follow in the footsteps of C and Emacs. And the principle of translating qualified package (or module, or class) names to file names is quite pervasive. Ada is an exception, rather than the rule. > I would provide a special syntax to handle this in > project-add-search-path: > > (defun project-add-search-path (project path &optional make-recursive) > "Add PATH (a list of directories) to the search path for PROJECT. > > If a directory in PATH ends in \"/**\", also add all directories under > that directory. > > If MAKE-RECURSIVE is non-nil, the full project search path is pruned so > that only highest-level directories are present, and all uses of > `project-search-path' recurse into subdirectories. In this case, it is > likely you will also need to specify ignored directories with > `project-add-ignores'. This description explains the purpose, but doesn't describe the resulting data structure that would hold that information. The ones that come to mind are rather ugly. > If MAKE-RECURSIVE is nil, uses of `project-search-path' no not recurse > into subdirectories." > ... ) > > The Ada project files support the ** sytnax. The /** syntax is rather nifty. But why do you propose to have it and MAKE-RECURSIVE at the same time? We could just interpret /** as the recursive marker. Then the result could be a list of strings. > This could be handled if the user can explicitly specify project ignores > (not just rely on .gitignore etc; git does _not_ ignore the lisp-emacs-* > and ada directories). More complicated structures are harder to handle > with ignores; you'll end up with a long flat list of ignores, so it > might be simpler to have a flat list of includes instead. I've just started to write about this... Yes indeed, I think every flat structure can be translated into path/to/dir, plus path/to/dir/*/ in the ignores. The translation function shouldn't take too much work to write. There's no need to produce the list of ignores manually. The Ada project backend could do that for the user. And if the list is long... first, you can use wildcards. Second, at least xref-collect-matches won't have to handle the flat case specially. Nor will the other consumers of the project API. > It would be surprising to an Ada user (used to flat search paths), if the > Emacs project silently started using recursive search paths. The Ada project implementation can provide a friendlier interface/set of customizations/etc for Ada users. Different communities are used to different things. Any default would be surprising to someone. >>> It is not at all clear how to convert a flat path into a recursive path >>> with ignores; the solution you have for elisp load-path is a kludge, >>> and could easily be wrong for some users. >> >> I'd like to see one of those users first. > > Me, and many other teams at NASA - structures similar to the above, > where the non-included directories can also be release version, > debug/release options, target hardware/os option, compiler > vendor/version, etc. There can be different opinions about this. As a project maintainer, I'd rather be able to find all references to the function foo, no matter the target OS of the code, or the targeted compiler version. > The jdee project has lisp and java directories. The Java class path is also "recursive". > The kludge is in the root implementation of project-search-path, so it > applies to all project backends that only customize > project-search-path-function. Please, stop calling major modes that do that "project backends". Customizing project-search-path-function does not constitute one. A real backend can always override project-search-path (and smart ones should). >>> So move it to "location" instead of "xref". The longest journey starts >>> with one step ... >> >> Until it's used in at least two different places, creating a separate >> package for it is, I think, premature. > > Right. We now have those two places; xref-find-definitions, > project-find-regexp. Both still use xref for display. The location values aren't consumed by anything else. So that's just one place. > Although it would be even better if project-find-regexp uses the compile > location data structure wrapped in a location-compile class, derived > from the root location class, renamed from xref-location. You're welcome to explore this direction with code. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-08-01 0:57 ` Dmitry Gutov @ 2015-08-01 9:50 ` Stephen Leake 2015-08-01 10:51 ` Stephen Leake 2015-08-01 12:40 ` Dmitry Gutov 0 siblings, 2 replies; 87+ messages in thread From: Stephen Leake @ 2015-08-01 9:50 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/31/2015 07:13 PM, Stephen Leake wrote: > >> Which is faster/friendlier depends on the details, so we'll have to wait >> for actual implementations. All I'm asking is that we do not make design >> decisions at the generic project API level that rule out this approach. > > We do have to make some decisions, though. For instance, supporting > both recursive and non-recursive elements in search-path will require > a separate code path for each function that uses it. > > As such, it will require a separate code path in xref-collect-matches > and an alternative for xref--rgrep-command. Or they could abort; that would be ok while we are experimenting. It may be that one or the other will turn out to be The Right Way. A backend that uses flat paths doesn't have to use those functions. Or, the people writing that backend can modify them if that seems better than putting similar code elsewhere. That would be part of the cost of that design decision. >> I would provide a special syntax to handle this in >> project-add-search-path: >> >> (defun project-add-search-path (project path &optional make-recursive) >> "Add PATH (a list of directories) to the search path for PROJECT. >> >> If a directory in PATH ends in \"/**\", also add all directories under >> that directory. >> >> If MAKE-RECURSIVE is non-nil, the full project search path is pruned so >> that only highest-level directories are present, and all uses of >> `project-search-path' recurse into subdirectories. In this case, it is >> likely you will also need to specify ignored directories with >> `project-add-ignores'. >> >> If MAKE-RECURSIVE is nil, uses of `project-search-path' no not recurse >> into subdirectories." >> ... ) >> >> The Ada project files support the ** sytnax. >> > This description explains the purpose, but doesn't describe the > resulting data structure that would hold that information. The ones > that come to mind are rather ugly. I have no idea what data structures come to mind for you. This could be implemented with a root projects type: (cl-defstruct (projects) user-search-path ;; list of directories added to project-search-path user-ignores ;; list of ignores added to project-ignores recursive-search-path ;; if non-nil, project-search-path is treated as recursive in all usees ... ) Is that "ugly"? > The /** syntax is rather nifty. But why do you propose to have it and > MAKE-RECURSIVE at the same time? We could just interpret /** as the > recursive marker. Then the result could be a list of strings. Because you never need both. If you are used to using recursive paths, adding ** in the path to tell the code that will seem odd. So it seems best to have an explicit flag. It might be better to specify the recursive flag in the project constructor; then this function would check it, not set it. >> This could be handled if the user can explicitly specify project ignores >> (not just rely on .gitignore etc; git does _not_ ignore the lisp-emacs-* >> and ada directories). More complicated structures are harder to handle >> with ignores; you'll end up with a long flat list of ignores, so it >> might be simpler to have a flat list of includes instead. > > I've just started to write about this... Yes indeed, I think every > flat structure can be translated into path/to/dir, plus path/to/dir/*/ > in the ignores. The translation function shouldn't take too much work > to write. The reverse is also true; it is possible to convert recursive path/ignores into flat path/ignores. > There's no need to produce the list of ignores manually. Only if you ignore the use cases I posted. There are situations where the algorithm will not do what the user wants. The fact that you don't like them does not mean they are not valid. Would it be such a problem to provide 'project-add-ignores'? >> Me, and many other teams at NASA - structures similar to the above, >> where the non-included directories can also be release version, >> debug/release options, target hardware/os option, compiler >> vendor/version, etc. > > There can be different opinions about this. Yes, precisely. And Emacs core should not be enforcing any opinions! >>>> So move it to "location" instead of "xref". The longest journey starts >>>> with one step ... >>> >>> Until it's used in at least two different places, creating a separate >>> package for it is, I think, premature. >> >> Right. We now have those two places; xref-find-definitions, >> project-find-regexp. > > Both still use xref for display. The location values aren't consumed > by anything else. So that's just one place. They both display locations. The code that displays locations should also be in the locations namespace, not the xref one. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-08-01 9:50 ` Stephen Leake @ 2015-08-01 10:51 ` Stephen Leake 2015-08-01 12:42 ` Dmitry Gutov 2015-08-01 12:40 ` Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-08-01 10:51 UTC (permalink / raw) To: emacs-devel Stephen Leake <stephen_leake@stephe-leake.org> writes: > Dmitry Gutov <dgutov@yandex.ru> writes: > >> The /** syntax is rather nifty. But why do you propose to have it and >> MAKE-RECURSIVE at the same time? We could just interpret /** as the >> recursive marker. Then the result could be a list of strings. I read this wrong. You _only_ need /** if the paths are _not_ recursive, so this cannot serve as the recursive flag. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-08-01 10:51 ` Stephen Leake @ 2015-08-01 12:42 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-08-01 12:42 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 08/01/2015 01:51 PM, Stephen Leake wrote: > You _only_ need /** if the paths are _not_ recursive, so this cannot > serve as the recursive flag. It would serve as "recursive flag" for each particular directory. Of course, in this approach the directories are flat by default. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-08-01 9:50 ` Stephen Leake 2015-08-01 10:51 ` Stephen Leake @ 2015-08-01 12:40 ` Dmitry Gutov 2015-08-01 14:15 ` Stephen Leake 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-08-01 12:40 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 08/01/2015 12:50 PM, Stephen Leake wrote: >> As such, it will require a separate code path in xref-collect-matches >> and an alternative for xref--rgrep-command. > > Or they could abort; that would be ok while we are experimenting. It may > be that one or the other will turn out to be The Right Way. That's not helpful. Of course we want the API be straightforward and usable for as many callers as possible. It's the main goal. I'm not sure how The Right Way would supposedly present itself. > A backend that uses flat paths doesn't have to use those functions. Or, > the people writing that backend can modify them if that seems better > than putting similar code elsewhere. That would be part of the cost of > that design decision. It's a high cost. If the author of such backend intends to reimplement or rewrite every feature like that, they can just as well not bother with implementing the Project API. As you can see, there's not much code in project.el, so they won't miss much if they don't value interoperability. > This could be implemented with a root projects type: > > (cl-defstruct (projects) > user-search-path ;; list of directories added to project-search-path > user-ignores ;; list of ignores added to project-ignores > recursive-search-path > ;; if non-nil, project-search-path is treated as recursive in all usees > ... > ) That doesn't tell me what values project-search-path would return, for every kind of configuration. That's the important part. > Is that "ugly"? Depends on how you feel about double negatives. There should be just one way to express whether a given directory should be traversed recursively. (cl-defstruct (projects) flat-search-path recursive-search-path would be one option for the implementation, but what would it mean for the API? If we have both project-flat-search-path and project-recursive-search-path as accessors, there's high probability that some functions will only look up and handle one of them. That's bad. >> The /** syntax is rather nifty. But why do you propose to have it and >> MAKE-RECURSIVE at the same time? We could just interpret /** as the >> recursive marker. Then the result could be a list of strings. > > Because you never need both. If you are used to using recursive paths, > adding ** in the path to tell the code that will seem odd. So it seems > best to have an explicit flag. The user could specify the paths in any way they like, are used to, etc. What's important is what the project backend returns in project-search-path. >> I've just started to write about this... Yes indeed, I think every >> flat structure can be translated into path/to/dir, plus path/to/dir/*/ >> in the ignores. The translation function shouldn't take too much work >> to write. > > The reverse is also true; it is possible to convert recursive > path/ignores into flat path/ignores. Not without drawbacks. First, it's more computationally intensive (Emacs would have to traverse the actual directory tree). Second, like I mentioned before, that loses information: in particular, we won't know which directories were the root ones. >> There's no need to produce the list of ignores manually. > > Only if you ignore the use cases I posted. There are situations where > the algorithm will not do what the user wants. What algorithm? I don't see why you can't write an algorithm that will both handle flat directories via ignores, as well as honor the user's wishes. > The fact that you don't like them does not mean they are not valid. > > Would it be such a problem to provide 'project-add-ignores'? I don't understand why you would state this question here? It doesn't seem relevant. >>> Me, and many other teams at NASA - structures similar to the above, >>> where the non-included directories can also be release version, >>> debug/release options, target hardware/os option, compiler >>> vendor/version, etc. >> >> There can be different opinions about this. > > Yes, precisely. And Emacs core should not be enforcing any opinions! Err, some opinions are worth being enforced. But anyway, the Ada project backend can behave any way you wish, in this respect. >> Both still use xref for display. The location values aren't consumed >> by anything else. So that's just one place. > > They both display locations. The code that displays locations should > also be in the locations namespace, not the xref one. What's left for xref, then? Most of the code there deals with displaying xrefs in the buffer. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-08-01 12:40 ` Dmitry Gutov @ 2015-08-01 14:15 ` Stephen Leake 2015-08-01 15:09 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-08-01 14:15 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 08/01/2015 12:50 PM, Stephen Leake wrote: > >> This could be implemented with a root projects type: >> >> (cl-defstruct (projects) >> user-search-path ;; list of directories added to project-search-path >> user-ignores ;; list of ignores added to project-ignores >> recursive-search-path >> ;; if non-nil, project-search-path is treated as recursive in all usees >> ... >> ) > > That doesn't tell me what values project-search-path would return, for > every kind of configuration. That's the important part. (cl-defgeneric project-search-path (project) "..." (project--prune-directories (nconc (project-user-search-path project) (funcall project-search-path-function) ;; Include these, because we don't know any better. ;; But a specialized implementation may include only some of ;; the project's subdirectories, if there are no source ;; files at the top level. (project-directories project)))) >> Is that "ugly"? > > Depends on how you feel about double negatives. There should be just > one way to express whether a given directory should be traversed > recursively. There is; the boolean flag recursive-search-path. > (cl-defstruct (projects) > flat-search-path > recursive-search-path > I'm guessing that here "flat-search-path" is a list of directories, intended to be used non-recursively. A comment would really help; guessing is dangerous in this discussion. And then "recursive-search-path" is a list of directories, intended to be used recursively. So they contain redundant information; they are each a cache of the result of the current `project-search-path' (with and without pruning). > would be one option for the implementation, but what would it mean for > the API? If we have both project-flat-search-path and > project-recursive-search-path as accessors, there's high probability > that some functions will only look up and handle one of them. That's > bad. Yes, it would be nice if we could agree that either flat or recursive is the baseline; any function that needs the other can convert. So far, I think we have these arguments on each side: - elisp load-path is already mostly flat it needs to be expanded to handle path/file style requires - Ada project paths are flat This is actually AdaCore project paths; they support C++ and C as well. - Some other project paths are recursive - Converting between them (while also converting the corresponding includes) is possible - Using find-grep and similar external programs allows recursive search to display first results immediately >>> I've just started to write about this... Yes indeed, I think every >>> flat structure can be translated into path/to/dir, plus path/to/dir/*/ >>> in the ignores. The translation function shouldn't take too much work >>> to write. >> >> The reverse is also true; it is possible to convert recursive >> path/ignores into flat path/ignores. > > Not without drawbacks. First, it's more computationally intensive > (Emacs would have to traverse the actual directory tree). You have to traverse the tree at some point; either during the use of the recursive path, or to convert the path. Converting it once is cheaper overall. But, as I've said before, this is an optimization issue. If it turns out that the performance either way is acceptable, it's not important. > Second, like > I mentioned before, that loses information: in particular, we won't > know which directories were the root ones. If that is useful information, obviously it can be saved. That's what project-roots is for. >>> There's no need to produce the list of ignores manually. >> >> Only if you ignore the use cases I posted. There are situations where >> the algorithm will not do what the user wants. > > What algorithm? The algorithm that automatically produces the list of ignores. > I don't see why you can't write an algorithm that will > both handle flat directories via ignores, as well as honor the user's > wishes. The user has to have a way of expressing those wishes. That's what project-add-ignores is for. >> The fact that you don't like them does not mean they are not valid. >> >> Would it be such a problem to provide 'project-add-ignores'? > > I don't understand why you would state this question here? It doesn't > seem relevant. I'll take that as agreeing to add the functions project-add-search-path and project-add-ignores to project.el Given all of the above, I'll make an attempt to write Ada and JDEE project backends, using recursive paths. We'll see how that goes. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-08-01 14:15 ` Stephen Leake @ 2015-08-01 15:09 ` Dmitry Gutov 2015-08-01 19:04 ` Stephen Leake 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-08-01 15:09 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 08/01/2015 05:15 PM, Stephen Leake wrote: > (cl-defgeneric project-search-path (project) > "..." > (project--prune-directories > (nconc (project-user-search-path project) > (funcall project-search-path-function) > ;; Include these, because we don't know any better. > ;; But a specialized implementation may include only some of > ;; the project's subdirectories, if there are no source > ;; files at the top level. > (project-directories project)))) That doesn't use recursive-search-path, and doesn't allow to treat only some elements recursively. Which would be natural. And if we allowed /**, the recursive-search-path flag would be redundant. >> (cl-defstruct (projects) >> flat-search-path >> recursive-search-path >> > > I'm guessing that here "flat-search-path" is a list of directories, > intended to be used non-recursively. > ... > And then "recursive-search-path" is a list of directories, intended to > be used recursively. Yes. > So they contain redundant information; they are each a cache of the > result of the current `project-search-path' (with and without pruning). This is backwards. There wouldn't be project-search-path function at all, and neither flat-search-path, nor recursive-search-path would be the result of pruning. > Yes, it would be nice if we could agree that either flat or recursive is > the baseline; any function that needs the other can convert. That's the direction I'm inclined to take, too. > So far, I think we have these arguments on each side: > > - elisp load-path is already mostly flat For out purposes, it's plainly recursive, even if that feature isn't in use in most projects. That's highly unlikely to ever change. > it needs to be expanded to handle path/file style requires Same as any other language with "recursive" search path that I know of. > - Ada project paths are flat > > This is actually AdaCore project paths; they support C++ and C as well. > > - Some other project paths are recursive Most of them, AFAIK. I've given several examples; there have been no further counter-examples mentioned. > - Converting between them (while also converting the corresponding > includes) is possible I think you mean the corresponding ignores. And converting a certain class of ignores from recursive to flat paths seems non-trivial. > - Using find-grep and similar external programs allows recursive search > to display first results immediately That's a good point, and I think it hasn't been stated clearly before. >> Second, like >> I mentioned before, that loses information: in particular, we won't >> know which directories were the root ones. > > If that is useful information, obviously it can be saved. That's what > project-roots is for. project-roots are different. This would have to be a separate accessor, because the search path roots outside of the project are just as (if not more) important. >> I don't see why you can't write an algorithm that will >> both handle flat directories via ignores, as well as honor the user's >> wishes. > > The user has to have a way of expressing those wishes. That's what > project-add-ignores is for. The user will express their wishes via the project file and .gitignore. There's nothing to discuss there; obviously, a project implementation will have access to all information the user will give it. > I'll take that as agreeing to add the functions project-add-search-path > and project-add-ignores to project.el Not really. I don't think we should mess with a project backend's internals. So unless we see a use case for third-party code to modify project's search-path and ignores, these functions should stay out of the API. > Given all of the above, I'll make an attempt to write Ada and JDEE > project backends, using recursive paths. We'll see how that goes. Hopefully, you'll make that attempt anyway. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-08-01 15:09 ` Dmitry Gutov @ 2015-08-01 19:04 ` Stephen Leake 2015-08-01 22:33 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-08-01 19:04 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: >> I'll take that as agreeing to add the functions project-add-search-path >> and project-add-ignores to project.el > > Not really. I don't think we should mess with a project backend's > internals. Well, yes; these should not be user level functions. But it would be nice if any backend could use them to build the project object. As you keep saying, "elisp does not have a project backend", so the user is forced to kludges for elisp projects. >> Given all of the above, I'll make an attempt to write Ada and JDEE >> project backends, using recursive paths. We'll see how that goes. > > Hopefully, you'll make that attempt anyway. Yes. I'll write what I need, and if there are functions/data structures used in both JDEE and Ada project backends, I'll propose them as additions to the project API. It turns out I'm also writing a more complete elisp backend as well. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-08-01 19:04 ` Stephen Leake @ 2015-08-01 22:33 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-08-01 22:33 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 08/01/2015 10:04 PM, Stephen Leake wrote: > Yes. I'll write what I need, and if there are functions/data structures > used in both JDEE and Ada project backends, I'll propose them as > additions to the project API. Great. That should be the way to go. Please be advised: I've added a second argument to project-ignores, like discussed before. > It turns out I'm also writing a more complete elisp backend as well. Maybe we could add it to Emacs as well. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Per-language project-search-path, was: Re: Unified project interface 2015-07-31 16:13 ` Stephen Leake 2015-08-01 0:57 ` Dmitry Gutov @ 2015-08-01 1:14 ` Dmitry Gutov 2015-08-01 10:43 ` Stephen Leake 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-08-01 1:14 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/31/2015 07:13 PM, Stephen Leake wrote: > The elisp project (depending on current Emacs version) will only want to > search one of: > > (lisp) > (lisp lisp-emacs-24) > (lisp lisp-emacs-23) > > The Ada project will only want to search the "ada" and "build" > directories (for the Makefile and Ada project file). By the way, do you think project-search-path should have a major-mode (or language) argument? Or maybe depend on it implicitly, via the current buffer? ede-source-paths dispatches on both PROJECT and MODE; to reuse it (in the ede project backend), we'd have to produce the mode argument. The caveats with the given choices are: - Users can use different major modes for the same files. Normally, project support shouldn't affect that. - We have no registry of languages, AFAIK. And different languages can be spelled differently (C++ vs Cpp, Fortran vs F77, Objective-C vs ObjC). - Just prescribing to use the current buffer in whatever way the project backend wants could be the most flexible, but it doesn't solve either difficulty. We could ask a similar question for project-ignores as well, though I'd rather not go that far. It needs a DIRECTORY argument already. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-01 1:14 ` Per-language project-search-path, was: " Dmitry Gutov @ 2015-08-01 10:43 ` Stephen Leake 2015-08-01 14:12 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-08-01 10:43 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 07/31/2015 07:13 PM, Stephen Leake wrote: > >> The elisp project (depending on current Emacs version) will only want to >> search one of: >> >> (lisp) >> (lisp lisp-emacs-24) >> (lisp lisp-emacs-23) >> >> The Ada project will only want to search the "ada" and "build" >> directories (for the Makefile and Ada project file). > > By the way, do you think project-search-path should have a major-mode > (or language) argument? Or maybe depend on it implicitly, via the > current buffer? My experience is always with a project file; the user explicitly selects which project file is active. That can be multi-language or single-language. So I'm not familiar with the project system trying to guess a project on the fly, as the current code does. As you point out, there are use cases for using the major mode to influence the project, and others for not doing that. I don't know which would be better as the default; the only general solution is to require a project file. > ede-source-paths dispatches on both PROJECT and MODE; to reuse it (in > the ede project backend), we'd have to produce the mode argument. Ede also uses the (require 'path/file) style; not surprising, since it was written by the CEDET team. I don't see any implementations of ede-source-paths, and only one use of it, so it's hard to say what the purpose of the mode parameter is. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-01 10:43 ` Stephen Leake @ 2015-08-01 14:12 ` Dmitry Gutov 2015-08-01 18:57 ` Stephen Leake 2015-08-02 2:29 ` Eric Ludlam 0 siblings, 2 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-08-01 14:12 UTC (permalink / raw) To: Stephen Leake, emacs-devel; +Cc: Eric Ludlam On 08/01/2015 01:43 PM, Stephen Leake wrote: > My experience is always with a project file; the user explicitly selects > which project file is active. That can be multi-language or > single-language. > > So I'm not familiar with the project system trying to guess a project on > the fly, as the current code does. I'm not discussing a particular project backend here, just the API. And you can use the user-selected backend everywhere, if there's an appropriate element at the head of project-find-functions. If I recall, you've mentioned a concept of project file listing different search-paths for different languages. Without project-search-path taking the language into account, you'll have to append all paths together. And what about jdee? Would you treat lisp/.dir-locals.el as the project file for Elisp, and ask the user to select it? > As you point out, there are use cases for using the major mode to > influence the project, and others for not doing that. I've only listed the implementation difficulties. If there are any particular problems with the concept of dispatching on the language, I'd like to hear about those. > I don't know which would be better as the default; the only general > solution is to require a project file. How does that translate to API requirements? > Ede also uses the (require 'path/file) style; not surprising, since it > was written by the CEDET team. It does. This doesn't seem relevant. > I don't see any implementations of ede-source-paths, and only one use of > it, so it's hard to say what the purpose of the mode parameter is. That's true, unfortunately. The only implementations I could find are in malabar-mode (another third-party project attempting to provide a Java editing environment). Eric, could you tell if we're missing anything? Are there other places where ede-source-paths is defined or used? Any reason why symref isn't using it? ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-01 14:12 ` Dmitry Gutov @ 2015-08-01 18:57 ` Stephen Leake 2015-08-02 0:25 ` Dmitry Gutov 2015-08-02 2:29 ` Eric Ludlam 1 sibling, 1 reply; 87+ messages in thread From: Stephen Leake @ 2015-08-01 18:57 UTC (permalink / raw) To: emacs-devel Dmitry Gutov <dgutov@yandex.ru> writes: > On 08/01/2015 01:43 PM, Stephen Leake wrote: > > If I recall, you've mentioned a concept of project file listing > different search-paths for different languages. Without > project-search-path taking the language into account, you'll have to > append all paths together. I don't know what you mean by "all paths" here; one project file defines one source path. In the case of a multilanguage project, that search path includes files in all languages. I often mix languages in one directory anyway; the file extension indicates the language, so there's no need for it in the directory name. > And what about jdee? Would you treat lisp/.dir-locals.el as the > project file for Elisp, and ask the user to select it? I think you are asking "what project would you use while editing elisp files that implement jdee" not "what project file does jdee use for Java projects". Yes, serious users will want to explicitly select the active project. If I'm debugging jdee, I want the search path to be elisp load-path, whether I'm in a notes.text file, a jdee .el file, or a test .java file. So I don't want Emacs to guess what the active project is, nor determine the search path based on the mode. When I'm testing jdee's use of the project API, I will explicitly select a Java project as the active project. I've just written this: (cl-defstruct (elisp-project) ;; no slots yet ) (cl-defmethod project-search-path ((type elisp-project)) (project--prune-directories load-path)) (cl-defmethod project-root ((type elisp-project)) ;; no meaningful root nil) ;; more overrides to come (setq project-explicit-prj (make-elisp-project)) See below for project-explicit-prj. In my mind, there is only one elisp project; all files currently on load-path. There will be a similar cl-defstruct for jdee-project. >> I don't know which would be better as the default; the only general >> solution is to require a project file. > > How does that translate to API requirements? - Provide a mechanism for the user to explicitly set a project file. For example (now in my local project-patches.el): (defvar project-explicit-prj nil "Project explicitly set by user.") (defun project-explicit-prj (dir) "Return the project the user has set in `project-explicit-prj'." project-explicit-prj) (setq project-find-functions 'project-explicit-prj) A menu or completion list of previously selected projects would be very useful here (I'm working on that). - Provide a way for functions to require a particular type of project file This is mostly there; all functions that only work with projects of some type should be cl-defmethods for that type. -- -- Stephe ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-01 18:57 ` Stephen Leake @ 2015-08-02 0:25 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-08-02 0:25 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 08/01/2015 09:57 PM, Stephen Leake wrote: > I often mix languages in one directory anyway; the file extension > indicates the language, so there's no need for it in the directory name. All right. That's fine if you don't need it. >> And what about jdee? Would you treat lisp/.dir-locals.el as the >> project file for Elisp, and ask the user to select it? > > I think you are asking > > "what project would you use while editing elisp files that implement > jdee" > > ... > > Yes, serious users will want to explicitly select the active project. Okay. I could add that different languages can have interdependencies (like templates can references to "normal" source code), so it's not always easy to separate them. But I guess you can use their search paths together, and rely on filtering by extension. > If I'm debugging jdee, I want the search path to be elisp load-path, > whether I'm in a notes.text file, a jdee .el file, or a test .java file. > So I don't want Emacs to guess what the active project is, nor determine > the search path based on the mode. Fair enough. > (setq project-find-functions 'project-explicit-prj) When the package is ready, this should be added via add-hook, when a certain minor mode is enabled. > > A menu or completion list of previously selected projects would be > very useful here (I'm working on that). That minor mode could provide a relevant command. > - Provide a way for functions to require a particular type of project file > > This is mostly there; all functions that only work with projects of > some type should be cl-defmethods for that type. That's fine, but it's better is more functions are universal. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-01 14:12 ` Dmitry Gutov 2015-08-01 18:57 ` Stephen Leake @ 2015-08-02 2:29 ` Eric Ludlam 2015-08-02 8:57 ` Nix 2015-08-03 1:21 ` Dmitry Gutov 1 sibling, 2 replies; 87+ messages in thread From: Eric Ludlam @ 2015-08-02 2:29 UTC (permalink / raw) To: Dmitry Gutov, Stephen Leake, emacs-devel; +Cc: Eric Ludlam Hi, Sorry for a long silence- it was a busy month. ;) On 08/01/2015 10:12 AM, Dmitry Gutov wrote: > On 08/01/2015 01:43 PM, Stephen Leake wrote: > >> My experience is always with a project file; the user explicitly selects >> which project file is active. That can be multi-language or >> single-language. >> >> So I'm not familiar with the project system trying to guess a project on >> the fly, as the current code does. > > I'm not discussing a particular project backend here, just the API. > And you can use the user-selected backend everywhere, if there's an > appropriate element at the head of project-find-functions. WRT EDE as an existing model to consider ... EDE will detect projects based on a key file, or you can ask it to load a project file with the ede command. It defines a base implementation of what a project is for handling menus and keybindings, and depends on different project backends to implement features used by the menus and keybindings. EDE does not have a list of 'find' type functions. Instead, each project type declares a small structure with functions for identifying the dominating file, which classes to implement, and for some complex cases, how to identify where projects are based on the contents of some config file. For example, Arduino has a config file that says where your sketches are. I used the structures because the allow project detection without autoloading the supporting back-end until that project type is actually used. > If I recall, you've mentioned a concept of project file listing > different search-paths for different languages. Without > project-search-path taking the language into account, you'll have to > append all paths together. EDE was not originally intended to be a way to search for files, but this feature was needed for semantic to find headers and referenced files using short names. It does this with 'ede-expand-filename'. For projects that include an exhaustive list of files as part of the configuration (such as Automake) searches just go through the known files using major-mode to filter if needed. Other projects that do not have all the files ready, such as EDE's android backend instead implement a method for `ede-expand-filename' that indicates where to start searching based on simple patterns, such as the java code in src, and resource xml files in res. Using methods is important so backend authors can do checking before blindly creating path names. For particularly large projects with a lot of source files, you can bind in an external tools such as Global or locate to help you find files when the basics don't work out. > I've only listed the implementation difficulties. If there are any > particular problems with the concept of dispatching on the language, > I'd like to hear about those. EDE had to manage the concept of derived modes via the mode-local tool in order to handle cases where C++ was like C with a few more details. If you intend to dispatch behavior based on mode, I strongly recommend mode local as a way to make it declarative and traceable. > >> I don't see any implementations of ede-source-paths, and only one use of >> it, so it's hard to say what the purpose of the mode parameter is. > > That's true, unfortunately. The only implementations I could find are > in malabar-mode (another third-party project attempting to provide a > Java editing environment). > > Eric, could you tell if we're missing anything? Are there other places > where ede-source-paths is defined or used? ede-source-paths is implemented for several java flavors, such as ant/maven and android. It is then used by semantic's javap database backend, and SRecode's template based code generator. It works well for java because class names and path names line up nicely. It hasn't been used for anything else, even though the comment mentions C++. I suspect what your source-paths concept better matches EDE's targets concept. EDE breaks a project into targets, matching a Makefile style target if that metadata is available. Each target represents some set of source files, usually all of the same mode. EDE also tracks include paths, which are locations where you might include code from some other location, matching the C/C++ concept. It works ok for some other languages, but primarily it is for that. Java has it's own classpath notion to better match what it does. > Any reason why symref isn't using it? Symref style searching, which I assume is like your xref concept, doesn't line up with the various paths in EDE, as EDE doesn't really need those types of paths for anything. Even the semantic tool, which used to deal with lists of project roots dropped the concept in favor of EDE's project detection as it was superior for all use cases. Every EDE project marks the top directory of a source tree, with .ede-ignore enabling breaks for nested projects. Using that top path, you can use find/grep, GNU Global, and various other tools to do your searching for you. The nice thing when you start using Global or CScope is that your config for the tool in question does the mode type filtering for you. My experience with EDE and other CEDET pieces is that I need to do symref type searches quite rarely. Usually the various tag based searches do a fine job getting me the data I need, the exception being finding uses of some function. In that case, symref, along with Global or cscope is nice because it does a good job organizing the results so it is easy to find what you want. I had intended symref to be where various refactoring tools would be based, as it allows you to select various hits, and perform operations on only selected hits. I stopped developing code as my job before I got to those features. Eric ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-02 2:29 ` Eric Ludlam @ 2015-08-02 8:57 ` Nix 2015-08-02 17:14 ` Michael Heerdegen 2015-08-02 23:07 ` Dmitry Gutov 2015-08-03 1:21 ` Dmitry Gutov 1 sibling, 2 replies; 87+ messages in thread From: Nix @ 2015-08-02 8:57 UTC (permalink / raw) To: Eric Ludlam; +Cc: michael_heerdegen, emacs-devel, Stephen Leake, Dmitry Gutov On 2 Aug 2015, Eric Ludlam stated: > Every EDE project marks the top directory of a source tree, with > .ede-ignore enabling breaks for nested projects. Using that top path, > you can use find/grep, GNU Global, and various other tools to do your > searching for you. The nice thing when you start using Global or > CScope is that your config for the tool in question does the mode type > filtering for you. Allow me to interject here that this feature had me applauding when I discovered it. It's an example of helpful automagic: if you have a large project, you're going to want a tagging system anyway: having Semantic automatically exploit it as soon as it appears is just gravy. Speaking personally, I don't need an extra project system: as far as I'm concerned Emacs already has one. I wrote one, as everyone does, and then threw it away because I never use it. EDE and GNU GLOBAL are sufficient, once you figure out how to use them: they know what files are part of the project, how to build it, how to find things in it with a degree of context-sensitivity that xref can only dream of (though its multi-backend feature means that presumably it can learn to *use* Semantic to provide context-sensitive search), and can handle things like projects with handwritten Makefiles that don't match its automagic project-root- searching features easily: (ede-cpp-root-project "DTrace-CTF" :file "/usr/src/libdtrace-ctf/Makerules" :include-path '("/include" "/build") :system-include-path '("/usr/include") :spp-table '(("_LITTLE_ENDIAN" . "") ("_GNU_SOURCE" . ""))) That, combined with a GTAGS file, gives me most of EDE's features, and all the features proposed for the project system. For projects like Emacs or the Linux kernel or anything using Automake I don't even need that much. EDE is an underappreciated jewel. Most of this project discussion and the invasive code piling into modes all across Emacs feels to me like an attempt to reimplement part of EDE, but without things like the grammar-sensitive searching that makes EDE so capable. (What I'd really like is something like the el-search pcase-based searcher Michael Heerdegen recently proposed, only for everything supported by Semantic, but with most language grammars not being very amenable to ML-style pattern-matching I doubt this is possible...) -- NULL && (void) ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-02 8:57 ` Nix @ 2015-08-02 17:14 ` Michael Heerdegen 2015-08-02 23:09 ` Eric Ludlam 2015-08-02 23:07 ` Dmitry Gutov 1 sibling, 1 reply; 87+ messages in thread From: Michael Heerdegen @ 2015-08-02 17:14 UTC (permalink / raw) To: Nix; +Cc: Eric Ludlam, emacs-devel, Stephen Leake, Dmitry Gutov Nix <nix@esperi.org.uk> writes: > (What I'd really like is something like the el-search pcase-based > searcher Michael Heerdegen recently proposed, only for everything > supported by Semantic, but with most language grammars not being very > amenable to ML-style pattern-matching I doubt this is possible...) I agree that would be very useful. But I don't know much about Semantic, and the info pages seem to be outdated. Is there some function in semantic that parses the expression at point and returns a parse tree represented as (Lisp) list? The outdated info page mentions ast.el which seems to do that, but I can't find it anymore. I guess one could try the following, more or less: go through the buffer and parse it on the fly to ast expressions. The search command prompts for a "pattern" using the syntax of the buffer's language plus placeholders. The input is parsed and converted to a an ast as well. Then the matching could be done in Elisp like el-search does. Michael. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-02 17:14 ` Michael Heerdegen @ 2015-08-02 23:09 ` Eric Ludlam 2015-08-02 23:39 ` Michael Heerdegen 0 siblings, 1 reply; 87+ messages in thread From: Eric Ludlam @ 2015-08-02 23:09 UTC (permalink / raw) To: Michael Heerdegen, Nix; +Cc: emacs-devel, Stephen Leake, Dmitry Gutov On 08/02/2015 01:14 PM, Michael Heerdegen wrote: > Nix <nix@esperi.org.uk> writes: > >> (What I'd really like is something like the el-search pcase-based >> searcher Michael Heerdegen recently proposed, only for everything >> supported by Semantic, but with most language grammars not being very >> amenable to ML-style pattern-matching I doubt this is possible...) > I agree that would be very useful. But I don't know much about > Semantic, and the info pages seem to be outdated. Most of the doc is about using some of the user facing tools. The sections about the lex stage should be good. > Is there some function in semantic that parses the expression at point > and returns a parse tree represented as (Lisp) list? The outdated info > page mentions ast.el which seems to do that, but I can't find it > anymore. > > I guess one could try the following, more or less: go through the buffer > and parse it on the fly to ast expressions. The search command prompts > for a "pattern" using the syntax of the buffer's language plus > placeholders. The input is parsed and converted to a an ast as well. > Then the matching could be done in Elisp like el-search does. You can use the lexer from semantic to get the list of parsed tokens. The current parsers output tags instead of an AST. Making an AST would be handy for langauge sensitive tooling past tagging parsers, but would be a bit slow. Large parts of the buffer aren't parsed at all right now as the parser skips over large { bracketed } blocks of code which keeps thing faster. It may be that the specifics you are looking for are part of the local context parser, which includes functions such as -get-local-variables, -get-local-arguments, and -current-symbol, which gets the whole symbols in a dotted expression. This is in semantic/ctxt.el. Eric ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-02 23:09 ` Eric Ludlam @ 2015-08-02 23:39 ` Michael Heerdegen 2015-08-03 11:33 ` Eric Ludlam 0 siblings, 1 reply; 87+ messages in thread From: Michael Heerdegen @ 2015-08-02 23:39 UTC (permalink / raw) To: Eric Ludlam; +Cc: Nix, emacs-devel, Stephen Leake, Dmitry Gutov Eric Ludlam <ericludlam@gmail.com> writes: > It may be that the specifics you are looking for are part of the local > context parser, which includes functions such as -get-local-variables, > -get-local-arguments, and -current-symbol, which gets the whole > symbols in a dotted expression. This is in semantic/ctxt.el. Thanks for the explanations. I gave these functions a quick try, evaluating them via M-: (the-function), with not much luck. In Elisp, all calls always returned nil. In a C file, I even got an error: Debugger entered--Lisp error: (wrong-type-argument syntax-table-p nil) set-syntax-table(nil) (progn (set-syntax-table semantic-lex-syntax-table) (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and (condition-case nil (progn ... ... t) (error nil)) (looking-at fieldsep1))) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons (buffer-substring-no-properties (point) end) symlist)) (let ((cp (point))) (forward-sexp -1) (forward-sexp 1) (if (<= cp (point)) (error nil))) (if (looking-at fieldsep) (progn (forward-sexp -1) (while (and ... ...) (forward-sexp -1)) (forward-sexp 1) (setq end (point))) (error nil))) (error nil))) symlist) (unwind-protect (progn (set-syntax-table semantic-lex-syntax-table) (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and (condition-case nil ... ...) (looking-at fieldsep1))) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons (buffer-substring-no-properties ... end) symlist)) (let ((cp ...)) (forward-sexp -1) (forward-sexp 1) (if (<= cp ...) (error nil))) (if (looking-at fieldsep) (progn (forward-sexp -1) (while ... ...) (forward-sexp 1) (setq end ...)) (error nil))) (error nil))) symlist) (save-current-buffer (set-buffer #:b uffer) (set-syntax-table #:table))) (let ((#2=#:table (syntax-table)) (#1=#:buffer (current-buffer))) (unwind-protect (progn (set-syntax-table semantic-lex-syntax-table) (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and ... ...)) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons ... symlist)) (let (...) (forward-sexp -1) (forward-sexp 1) (if ... ...)) (if (looking-at fieldsep) (progn ... ... ... ...) (error nil))) (error nil))) symlist) (save-current-buffer (set-buffer #1#) (set-syntax-table #2#)))) (with-syntax-table semantic-lex-syntax-table (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and (condition-case nil (progn ... ... t) (error nil)) (looking-at fieldsep1))) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons (buffer-substring-no-properties (point) end) symlist)) (let ((cp (point))) (forward-sexp -1) (forward-sexp 1) (if (<= cp (point)) (error nil))) (if (looking-at fieldsep) (progn (forward-sexp -1) (while (and ... ...) (forward-sexp -1)) (forward-sexp 1) (setq end (point))) (error nil))) (error nil))) symlis t) (let* ((fieldsep1 (mapconcat (lambda (a) (regexp-quote a)) semantic-type-relation-separator-character "\\|")) (fieldsep (concat "[ \n.]*\\(" fieldsep1 "\\)[ \n.]*\\(\\w\\|\\s_\\)")) (case-fold-search semantic-case-fold) (symlist nil) end) (with-syntax-table semantic-lex-syntax-table (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and (condition-case nil ... ...) (looking-at fieldsep1))) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons (buffer-substring-no-properties ... end) symlist)) (let ((cp ...)) (forward-sexp -1) (f orward-sexp 1) (if (<= cp ...) (error nil))) (if (looking-at fieldsep) (progn (forward-sexp -1) (while ... ...) (forward-sexp 1) (setq end ...)) (error nil))) (error nil))) symlist)) (save-excursion (if point (goto-char point)) (let* ((fieldsep1 (mapconcat (lambda (a) (regexp-quote a)) semantic-type-relation-separator-character "\\|")) (fieldsep (concat "[ \n.]*\\(" fieldsep1 "\\)[ \n.]*\\(\\w\\|\\s_\\)")) (case-fold-search semantic-case-fold) (symlist nil) end) (with-syntax-table semantic-lex-syntax-table (save-excursion (cond ((looking-at "\\w\\|\\s_") (forward-sexp 1)) ((looking-at fieldsep1) nil) ((save-excursion (and ... ...)) (setq symlist (list "")) (forward-sexp -1) (while (looking-at "\\s(") (forward-sexp -1)) (forward-sexp 1))) (setq end (point)) (condition-case nil (while (save-excursion (forward-char -1) (looking-at "\\w\\|\\s_")) (forward-sexp -1) (setq symlist (cons ... symlist)) (let (...) (forward-sexp -1) (forward-sexp 1) (if ... ...)) (if (looking -at fieldsep) (progn ... ... ... ...) (error nil))) (error nil))) symlist))) semantic-ctxt-current-symbol-default() ... because semantic-lex-syntax-table is bound to nil which `set-syntax-table' doesn't like. Michael. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-02 23:39 ` Michael Heerdegen @ 2015-08-03 11:33 ` Eric Ludlam 0 siblings, 0 replies; 87+ messages in thread From: Eric Ludlam @ 2015-08-03 11:33 UTC (permalink / raw) To: Michael Heerdegen; +Cc: Nix, emacs-devel, Stephen Leake, Dmitry Gutov On 08/02/2015 07:39 PM, Michael Heerdegen wrote: > Eric Ludlam <ericludlam@gmail.com> writes: > >> It may be that the specifics you are looking for are part of the local >> context parser, which includes functions such as -get-local-variables, >> -get-local-arguments, and -current-symbol, which gets the whole >> symbols in a dotted expression. This is in semantic/ctxt.el. > Thanks for the explanations. > > I gave these functions a quick try, evaluating them via M-: > (the-function), with not much luck. In Elisp, all calls always returned > nil. In a C file, I even got an error: > > Debugger entered--Lisp error: (wrong-type-argument syntax-table-p nil) > set-syntax-table(nil) > (progn (set-syntax-table semantic-lex-syntax-table) (save-excursion (cond ((looking-at "\\w\\|\\s_") > because semantic-lex-syntax-table is bound to nil which > `set-syntax-table' doesn't like. > Everything in the semantic system needs `semantic-mode' enabled. That sets up all the parsing context needed for the various tools, and keeps it up to date. There is a lot of little features needed to handle parsing, so the mode enables the hooks to get it setup as you load files, and then the tagging parser parsers you buffers, which the context parser uses to identify the outer bounds of different contexts. Once you do that, you could try 'semantic-analyze-current-context' which will give you some high level data about where point is. The local -ctxt- functions can be used to do something unrelated smart completion. Eric ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-02 8:57 ` Nix 2015-08-02 17:14 ` Michael Heerdegen @ 2015-08-02 23:07 ` Dmitry Gutov 2015-08-03 10:24 ` Nix 1 sibling, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-08-02 23:07 UTC (permalink / raw) To: Nix, Eric Ludlam; +Cc: michael_heerdegen, Stephen Leake, emacs-devel On 08/02/2015 11:57 AM, Nix wrote: > Allow me to interject here that this feature had me applauding when I > discovered it. It's an example of helpful automagic: if you have a large > project, you're going to want a tagging system anyway: having Semantic > automatically exploit it as soon as it appears is just gravy. If EDE works for you, that's great. You should continue using it. > Speaking personally, I don't need an extra project system: as far as I'm > concerned Emacs already has one. The point is other tools can use some information about your current project without being aware that you're using EDE. Or something else. > the project, how to build it, how to find things in it with a degree of > context-sensitivity that xref can only dream of (though its > multi-backend feature means that presumably it can learn to *use* > Semantic to provide context-sensitive search) Indeed, it can. *And* it's also useful for languages which Semantic doesn't support. > Most of this project > discussion and the invasive code piling into modes all across Emacs > feels to me like an attempt to reimplement part of EDE, but without > things like the grammar-sensitive searching that makes EDE so capable. What code piling? ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-02 23:07 ` Dmitry Gutov @ 2015-08-03 10:24 ` Nix 2015-08-03 10:35 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Nix @ 2015-08-03 10:24 UTC (permalink / raw) To: Dmitry Gutov; +Cc: michael_heerdegen, Eric Ludlam, Stephen Leake, emacs-devel On 3 Aug 2015, Dmitry Gutov spake thusly: > On 08/02/2015 11:57 AM, Nix wrote: >> Most of this project >> discussion and the invasive code piling into modes all across Emacs >> feels to me like an attempt to reimplement part of EDE, but without >> things like the grammar-sensitive searching that makes EDE so capable. > > What code piling? There seem to be rather a lot of references to project-* landing in unrelated files (language modes, etc). Or so it seems to me, anyway. (But it's much less invasive than, say, the curved-quotes stuff, and much more useful ;) ) -- NULL && (void) ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-03 10:24 ` Nix @ 2015-08-03 10:35 ` Dmitry Gutov 2015-08-07 15:25 ` Nix 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-08-03 10:35 UTC (permalink / raw) To: Nix; +Cc: michael_heerdegen, Eric Ludlam, Stephen Leake, emacs-devel On 08/03/2015 01:24 PM, Nix wrote: > There seem to be rather a lot of references to project-* landing in > unrelated files (language modes, etc). Or so it seems to me, anyway. ^ └ citation needed ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-03 10:35 ` Dmitry Gutov @ 2015-08-07 15:25 ` Nix 0 siblings, 0 replies; 87+ messages in thread From: Nix @ 2015-08-07 15:25 UTC (permalink / raw) To: Dmitry Gutov; +Cc: michael_heerdegen, Eric Ludlam, Stephen Leake, emacs-devel On 3 Aug 2015, Dmitry Gutov verbalised: > On 08/03/2015 01:24 PM, Nix wrote: > >> There seem to be rather a lot of references to project-* landing in >> unrelated files (language modes, etc). Or so it seems to me, anyway. > ^ > └ citation needed None available because it was predicated on a bunch of misunderstandings of how tightly things like xref tie into it. :) sorry! -- NULL && (void) ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Per-language project-search-path, was: Re: Unified project interface 2015-08-02 2:29 ` Eric Ludlam 2015-08-02 8:57 ` Nix @ 2015-08-03 1:21 ` Dmitry Gutov 1 sibling, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-08-03 1:21 UTC (permalink / raw) To: Eric Ludlam, Stephen Leake, emacs-devel; +Cc: Eric Ludlam Hi Erc, On 08/02/2015 05:29 AM, Eric Ludlam wrote: > EDE does not have a list of 'find' type functions. Instead, each > project type declares a small structure with functions for identifying > the dominating file, which classes to implement, and for some complex "which classes to implement" - what does that mean? Is this something interesting, in the context of this discussion? > EDE was not originally intended to be a way to search for files, but > this feature was needed for semantic to find headers and referenced > files using short names. I think it's quite natural for a project to be able to list the directories where related files could be found. For instance, so that someone could implement a find-file-in-project command. Or search across those files. > It does this with 'ede-expand-filename'. For > projects that include an exhaustive list of files as part of the > configuration (such as Automake) searches just go through the known > files using major-mode to filter if needed. This seems useful enough, although maybe I'd untie the expand-filename logic from a given project implementation and make it dependent on the major mode. For instance, no matter if it's a common Java project or an Android project, whether it's based on Ant, or Maven, or Gradle, the logic for finding the file based on the class name is the same: you just need the list of directories to look in. Which project-search-path might correspond to. > For particularly large projects with a lot of source files, you can bind > in an external tools such as Global or locate to help you find files > when the basics don't work out. Global seems fine to search inside the project (for certain languages, at least), but what about the system libraries? Headers, etc. If they're used in the current project, being able to search in them would be useful, too. > EDE had to manage the concept of derived modes via the mode-local tool > in order to handle cases where C++ was like C with a few more details. > If you intend to dispatch behavior based on mode, I strongly recommend > mode local as a way to make it declarative and traceable. That's interesting, but it doesn't sidestep the issue of different users possibly using alternative major modes for the same files. So far as I see it, in most cases the project implementation could easily write something ad-hoc like that that would be good enough. IOW, I'm more concerned about the basics right now. > ede-source-paths is implemented for several java flavors, such as > ant/maven and android. It is then used by semantic's javap database > backend, and SRecode's template based code generator. I see, thank you. I've only been looking in CEDET inside Emacs. > It works well for java because class names and path names line up > nicely. It hasn't been used for anything else, even though the comment > mentions C++. In Ruby, class and module names also often map to file paths. Only by convention, though, so there's some variability there. The same happens, to certain degrees, in other languages too. > I suspect what your source-paths concept better matches EDE's targets > concept. EDE breaks a project into targets, matching a Makefile style > target if that metadata is available. Each target represents some set > of source files, usually all of the same mode. Maybe? I'm not sure how the concept of targets maps to dynamic languages, for example. > EDE also tracks include paths, which are locations where you might > include code from some other location, matching the C/C++ concept. It > works ok for some other languages, but primarily it is for that. Java > has it's own classpath notion to better match what it does. Indeed. project-search-path, as it's currently imagined, is a merge of ede-source-paths and ede-system-include-path. So xref-find-regexp and xref-find-references will, by default, search the includes path too. Something that symref doesn't do. >> Any reason why symref isn't using it? > > Symref style searching, which I assume is like your xref concept, > doesn't line up with the various paths in EDE, as EDE doesn't really > need those types of paths for anything. Even the semantic tool, which > used to deal with lists of project roots dropped the concept in favor of > EDE's project detection as it was superior for all use cases. Yes, we also try to reuse semantic-symref-find-references-by-name in the default implementation of xref-find-references. The two main drawbacks now are: it doesn't handle the project's ignored directories (which should be relevant for symref/grep.el), and semantic-symref-filepattern-alist needs to have correct entries for every language. > Every EDE project marks the top directory of a source tree, with > .ede-ignore enabling breaks for nested projects. Using that top path, > you can use find/grep, GNU Global, and various other tools to do your > searching for you. The nice thing when you start using Global or CScope > is that your config for the tool in question does the mode type > filtering for you. I suppose. As long as the tool supports all kinds of operators (search for references, regexp, listing files), and supports your target language. Though it probably won't help with related code outside of your project's tree anyway. Maybe xref-find-references should also always search all project-roots anyway (while filtering by file names and ignoring non-interesting directories), and project-search-path should focus on "source roots" (inside and outside of the project tree) exclusively. So even if the project has a build file written in the same language as the rest of it (and it lies in the root directory), project-search-path still would only include, say, project/lib and project/app. The main use case for those entries, then, would be locating the file for a given class name. Which could serve as a basis for a naive find-definition implementation. > My experience with EDE and other CEDET pieces is that I need to do > symref type searches quite rarely. Usually the various tag based > searches do a fine job getting me the data I need, the exception being > finding uses of some function. "Finding uses" is an operation I have to do quite frequently. >I had intended symref to be where > various refactoring tools would be based, as it allows you to select > various hits, and perform operations on only selected hits. Yup, support for renaming is something I'm interested in as well. It's generally accepted that Emacs's refactoring support is lacking/nonexistent. ^ permalink raw reply [flat|nested] 87+ messages in thread
* xref display and multiple locations, was: Re: Unified project interface 2015-07-29 2:27 ` Stephen Leake 2015-07-29 22:51 ` Dmitry Gutov @ 2015-07-29 23:11 ` Dmitry Gutov 1 sibling, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-07-29 23:11 UTC (permalink / raw) To: Stephen Leake, emacs-devel On 07/29/2015 05:27 AM, Stephen Leake wrote: > Currently, 'grep-find' and 'compilation' share the same location API. > Which is why I suggested that xref also share it. But I do like the > current xref location display, so maybe there is a case for them to > remain distinct. > > They serve different purposes; the xref display makes it easy to choose > one location from a small list; the compilation display makes it easier > to step through all of them in sequence, and it works even for very long > lists. I've come to believe that it's a mistake. The only xref command where we're likely to only be only in one location is xref-find-definitions, and in ideal implementations, it's likely not to show the xref buffer at all, jumping straight to the sole result. So in the average case, pressing RET, or clicking on an xref element, should display the location, but keep the xref buffer displayed as well. We just need a way to use the current behavior in xref-find-definitions (or implement a yet-another selection interface for it). > There should at least there should be this choice of display; the > location data structure could probably be shared (the compilation > location structure contains more info). The current customization point is xref-show-xrefs-function. There are no different implementations yet. One nice feature of M-x grep, by the way, is that it shows the matches immediately as it finds them (whereas the current xref commands are forced to wait until the output ends, and collect the whole list). One way to reach parity there would be to teach xref to accept generators (as implemented in generator.el), as well as regular lists. > (grep-find could eventually be made obsolete in favor of > project-find-regexp). Here's hoping. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-04 11:43 Unified project interface Dmitry Gutov 2015-06-04 14:40 ` Stephen Leake @ 2015-06-06 10:20 ` Bozhidar Batsov 2015-06-06 10:29 ` Dmitry Gutov 2015-06-06 12:32 ` Eric Ludlam 2 siblings, 1 reply; 87+ messages in thread From: Bozhidar Batsov @ 2015-06-06 10:20 UTC (permalink / raw) To: Dmitry Gutov; +Cc: Jorgen Schaefer, emacs-devel [-- Attachment #1: Type: text/plain, Size: 1133 bytes --] Can you explain in simple terms what exactly are the use cases for the functionality? Being able to find definitions and grep in projects? I recall having some discussion a while back, but the details are really vague in my head. On 4 June 2015 at 13:43, Dmitry Gutov <dgutov@yandex.ru> wrote: > Hi all, > > A while ago, there was floated a "unified project root" proposal. I'd like > to resume that discussion. > > Unfortunately, the code proposed back them doesn't serve the immediate > need I have in mind: basically, I need that function to return a *list* of > directories, because both etags and elisp xref backends operate on multiple > directories, via tags-table-list and load-path respectively. > > If we have it, we'll be able to untie `xref-find-regexp' from xref > backends, because really there's nothing language-specific to this command. > > Now, before adding this feature to the core, are there other related > general questions we want to be able to answer? Depending on that, a > "project" might be better implemented as a cl-struct, providing specialized > implementations for those several generic methods. > [-- Attachment #2: Type: text/html, Size: 1479 bytes --] ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-06 10:20 ` Bozhidar Batsov @ 2015-06-06 10:29 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-06-06 10:29 UTC (permalink / raw) To: Bozhidar Batsov; +Cc: Jorgen Schaefer, emacs-devel On 06/06/2015 01:20 PM, Bozhidar Batsov wrote: > Can you explain in simple terms what exactly are the use cases for the > functionality? That's what this thread is about: to discuss different use cases for the unified project interface. > Being able to find definitions and grep in projects? I > recall having some discussion a while back, but the details are really > vague in my head. xref-find-regexp will use it (always, I think). xref-find-references will also use it, at least the default implementation. I haven't considered, thus far, creating a default xref-find-definitions implementation along these lines. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-04 11:43 Unified project interface Dmitry Gutov 2015-06-04 14:40 ` Stephen Leake 2015-06-06 10:20 ` Bozhidar Batsov @ 2015-06-06 12:32 ` Eric Ludlam 2015-06-06 18:44 ` Dmitry Gutov 2 siblings, 1 reply; 87+ messages in thread From: Eric Ludlam @ 2015-06-06 12:32 UTC (permalink / raw) To: Dmitry Gutov, emacs-devel; +Cc: Jorgen Schaefer, Bozhidar Batsov On 06/04/2015 07:43 AM, Dmitry Gutov wrote: > Hi all, > > A while ago, there was floated a "unified project root" proposal. I'd > like to resume that discussion. > [...] > > Now, before adding this feature to the core, are there other related > general questions we want to be able to answer? Depending on that, a > "project" might be better implemented as a cl-struct, providing > specialized implementations for those several generic methods. In that old thread, I had proposed using EDE either as a supplier to a generic interface, or as a way to build said interface. Several people pointed out flaws in EDE's project detection scheme. Since that time I have accounted for those recommendations and added a severe test suite for detecting projects more cleanly. Those changes were merged into Emacs a while back. As an aside, thanks for the advice back then. In general, if there is a proposed generic project root type I'll be happy to enabled EDE to supply data to it. Alternately, if there is an interest in an EDE-lite where key functionality such as detection and providing a root path is pulled up into something that doesn't turn on all the other unwanted EDE features, I'd be happy to advise or help. For an example, in 25.x, in cedet/ede/generic.el, look for `ede-enable-generic-projects' for some simple definitions. I'm sure the generic project type in EDE is overkill for what is wanted, so I'm assuming there would need to be a stripped down version. As a reminder, ede uses CLOS, so there is a base object representing the project, and you can run methods against it to get information which would be simple to expand to any type of desired generic interface. Eric ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-06 12:32 ` Eric Ludlam @ 2015-06-06 18:44 ` Dmitry Gutov 2015-06-06 19:28 ` Eli Zaretskii 0 siblings, 1 reply; 87+ messages in thread From: Dmitry Gutov @ 2015-06-06 18:44 UTC (permalink / raw) To: Eric Ludlam, emacs-devel; +Cc: Jorgen Schaefer, Bozhidar Batsov Hi Eric, On 06/06/2015 03:32 PM, Eric Ludlam wrote: > In general, if there is a proposed generic project root type I'll be > happy to enabled EDE to supply data to it. That's good. > Alternately, if there is an > interest in an EDE-lite where key functionality such as detection and > providing a root path is pulled up into something that doesn't turn on > all the other unwanted EDE features, I'd be happy to advise or help. > > For an example, in 25.x, in cedet/ede/generic.el, look for > `ede-enable-generic-projects' for some simple definitions. I'm sure the > generic project type in EDE is overkill for what is wanted, so I'm > assuming there would need to be a stripped down version. While it looks pretty decent, I'm among those'd prefer to see CEDET moved out to GNU ELPA. And anyway, it would be better to have the smallest possible API that, say, Projectile could implement without relying on EDE being present and loaded. But any lessons EDE can teach us about making that API better, would be great to have. > As a reminder, ede uses CLOS, so there is a base object representing the > project, and you can run methods against it to get information which > would be simple to expand to any type of desired generic interface. I imagine the new API will work along the same principle, only using cl-generic instead of eieio. Does EDE store information about paths outside of the current project root? Like linked projects, or C include directories, or load-path elements (if EDE supports Elisp projects)? Does it allow to quickly grep across all of them, or use semantic-symref, again, across all of them? ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-06 18:44 ` Dmitry Gutov @ 2015-06-06 19:28 ` Eli Zaretskii 2015-06-07 22:29 ` Dmitry Gutov 0 siblings, 1 reply; 87+ messages in thread From: Eli Zaretskii @ 2015-06-06 19:28 UTC (permalink / raw) To: Dmitry Gutov; +Cc: contact, emacs-devel, bozhidar, eric > From: Dmitry Gutov <dgutov@yandex.ru> > Date: Sat, 6 Jun 2015 21:44:12 +0300 > Cc: Jorgen Schaefer <contact@jorgenschaefer.de>, > Bozhidar Batsov <bozhidar@batsov.com> > > I'm among those'd prefer to see CEDET moved out to GNU ELPA. If we want the IDE capabilities of Emacs become better, this should never happen. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: Unified project interface 2015-06-06 19:28 ` Eli Zaretskii @ 2015-06-07 22:29 ` Dmitry Gutov 0 siblings, 0 replies; 87+ messages in thread From: Dmitry Gutov @ 2015-06-07 22:29 UTC (permalink / raw) To: Eli Zaretskii; +Cc: contact, emacs-devel, bozhidar, eric On 06/06/2015 10:28 PM, Eli Zaretskii wrote: > If we want the IDE capabilities of Emacs become better, this should > never happen. IME, CEDET is fairly interdependent, and Semantic's approach doesn't seem to be useful enough for, say, code completion in dynamic languages. I see the path forward as different CEDET features growing into more flexible APIs that different packages can implement. Like semantic-symref now has a generic counterpart of xref-find-references, and while the latter has a simplistic grep-based implementation, CEDET can define its own xref backend, to implement the latter through the former. ^ permalink raw reply [flat|nested] 87+ messages in thread
end of thread, other threads:[~2015-08-07 15:25 UTC | newest] Thread overview: 87+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-06-04 11:43 Unified project interface Dmitry Gutov 2015-06-04 14:40 ` Stephen Leake 2015-06-05 0:08 ` Dmitry Gutov 2015-06-05 10:08 ` Stephen Leake 2015-06-05 13:03 ` Stephen Leake 2015-06-05 13:14 ` Dmitry Gutov 2015-06-08 1:24 ` Stephen Leake 2015-06-09 18:16 ` Dmitry Gutov 2015-06-09 18:21 ` Eli Zaretskii 2015-06-09 18:49 ` Dmitry Gutov 2015-06-09 19:03 ` Eli Zaretskii 2015-06-07 23:22 ` Dmitry Gutov 2015-06-08 1:35 ` [SPAM UNSURE] " Stephen Leake 2015-06-09 19:04 ` Dmitry Gutov 2015-06-07 23:15 ` Dmitry Gutov 2015-06-08 1:59 ` Stephen Leake 2015-06-09 22:31 ` Dmitry Gutov 2015-06-10 7:13 ` Steinar Bang 2015-07-08 0:25 ` Dmitry Gutov 2015-07-11 13:43 ` Bozhidar Batsov 2015-07-11 14:17 ` Dmitry Gutov 2015-07-12 14:42 ` Dmitry Gutov 2015-07-13 8:49 ` Bozhidar Batsov 2015-07-13 10:23 ` Dmitry Gutov 2015-07-24 23:43 ` Dmitry Gutov 2015-07-25 0:55 ` Stephen Leake 2015-07-25 7:29 ` Eli Zaretskii 2015-07-26 2:12 ` Dmitry Gutov 2015-07-26 2:45 ` Eli Zaretskii 2015-07-26 11:25 ` Stephen Leake 2015-07-26 2:11 ` Dmitry Gutov 2015-07-26 11:22 ` Stephen Leake 2015-07-26 17:23 ` Dmitry Gutov 2015-07-26 18:57 ` Stephen Leake 2015-07-26 23:56 ` John Yates 2015-07-27 1:49 ` Dmitry Gutov 2015-07-27 11:12 ` Stephen Leake 2015-07-27 11:27 ` Dmitry Gutov 2015-07-27 13:00 ` Dmitry Gutov 2015-07-27 13:02 ` Dmitry Gutov 2015-07-28 1:21 ` Stephen Leake 2015-07-28 11:05 ` Stephen Leake 2015-07-28 14:33 ` Dmitry Gutov 2015-07-28 15:45 ` Stephen Leake 2015-07-28 16:25 ` Dmitry Gutov 2015-07-29 1:36 ` Stephen Leake 2015-07-29 2:10 ` Dmitry Gutov 2015-07-28 14:18 ` Dmitry Gutov 2015-07-28 16:15 ` Stephen Leake 2015-07-28 18:44 ` Dmitry Gutov 2015-07-29 2:27 ` Stephen Leake 2015-07-29 22:51 ` Dmitry Gutov 2015-07-30 8:17 ` Stephen Leake 2015-07-31 0:15 ` Dmitry Gutov 2015-07-31 16:13 ` Stephen Leake 2015-08-01 0:57 ` Dmitry Gutov 2015-08-01 9:50 ` Stephen Leake 2015-08-01 10:51 ` Stephen Leake 2015-08-01 12:42 ` Dmitry Gutov 2015-08-01 12:40 ` Dmitry Gutov 2015-08-01 14:15 ` Stephen Leake 2015-08-01 15:09 ` Dmitry Gutov 2015-08-01 19:04 ` Stephen Leake 2015-08-01 22:33 ` Dmitry Gutov 2015-08-01 1:14 ` Per-language project-search-path, was: " Dmitry Gutov 2015-08-01 10:43 ` Stephen Leake 2015-08-01 14:12 ` Dmitry Gutov 2015-08-01 18:57 ` Stephen Leake 2015-08-02 0:25 ` Dmitry Gutov 2015-08-02 2:29 ` Eric Ludlam 2015-08-02 8:57 ` Nix 2015-08-02 17:14 ` Michael Heerdegen 2015-08-02 23:09 ` Eric Ludlam 2015-08-02 23:39 ` Michael Heerdegen 2015-08-03 11:33 ` Eric Ludlam 2015-08-02 23:07 ` Dmitry Gutov 2015-08-03 10:24 ` Nix 2015-08-03 10:35 ` Dmitry Gutov 2015-08-07 15:25 ` Nix 2015-08-03 1:21 ` Dmitry Gutov 2015-07-29 23:11 ` xref display and multiple locations, " Dmitry Gutov 2015-06-06 10:20 ` Bozhidar Batsov 2015-06-06 10:29 ` Dmitry Gutov 2015-06-06 12:32 ` Eric Ludlam 2015-06-06 18:44 ` Dmitry Gutov 2015-06-06 19:28 ` Eli Zaretskii 2015-06-07 22:29 ` Dmitry Gutov
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.