bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 1896 bytes --]

This patch add support for "plain project" in project.el. Plain project is a
kind of project without any VC backend but should be.

To mark a directoy as project, put an empty magic file .emacs-project under
the
directory, and project.el should be responsible for it.

~~~~

From cb0a67cfacf141a8b1955c08c3f459bcac801a39 Mon Sep 17 00:00:00 2001
From: Zhu Zihao <all_but_last@163.com>
Date: Thu, 28 May 2020 11:04:44 +0800
Subject: [PATCH] Support plain project marked with file .emacs-project

* lisp/progmodes/project.el (project-try-plain): New function
* lisp/progmodes/project.el (project-root): New dispatch variants
* lisp/progmodes/project.el (project-find-functions): Add project-try-plain
---
 lisp/progmodes/project.el | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 88f73e4fb31..4c1810aeb56 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -94,7 +94,7 @@

 (require 'cl-generic)

-(defvar project-find-functions (list #'project-try-vc)
+(defvar project-find-functions (list #'project-try-plain #'project-try-vc)
   "Special hook to find the project containing a given directory.
 Each functions on this hook is called in turn with one
 argument (the directory) and should return either nil to mean
@@ -194,6 +194,18 @@ project-files
    (or dirs
        (list (project-root project)))))

+(defun project-try-plain (dir)
+  "Return the plain project instance of current DIR.
+
+A directory with magic file \".emacs-project\" will be recognized as
+plain project."
+  (pcase (locate-dominating-file dir ".emacs-project")
+    (`nil nil)
+    (root (cons 'plain root))))
+
+(cl-defmethod project-root ((project (head plain)))
+  (cdr project))
+
 (defun project--files-in-directory (dir ignores &optional files)
   (require 'find-dired)
   (require 'xref)
-- 
2.26.2

[-- Attachment #2: Type: text/html, Size: 2328 bytes --]

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Philip K.]

Zhu Zihao <cjpeople2013@gmail.com> writes:

> To mark a directoy as project, put an empty magic file .emacs-project under the
> directory, and project.el should be responsible for it.

Is there any more standard name than ".emacs-project"? Or could a
directory local-variable be used? I like the idea, but wouldn't want to
have so many ".emacs-project" files lying around in toy projects.

-- 
	Philip K.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Zihao Zhu]

IMO, it's not practical to use directory local variable

1. directory local variable goes ".dir-local.el". But we can't mark 
every directory contain this file as project. We have to do extra search 
if we use directory local variable.

2. If we have variable "project-directory-plain-project-p", It's a 
problem for us to determine the root of project because we will get the 
same directory local value for all sub-directories of project root.

In your use case. I think we can add variable "project-known-projects", 
you can  add your favorite directory to it, you can also persist this 
variable using savehist.el


On 2020/5/28 下午3:42, Philip K. wrote:
> Zhu Zihao <cjpeople2013@gmail.com> writes:
>
>> To mark a directoy as project, put an empty magic file .emacs-project under the
>> directory, and project.el should be responsible for it.
> Is there any more standard name than ".emacs-project"? Or could a
> directory local-variable be used? I like the idea, but wouldn't want to
> have so many ".emacs-project" files lying around in toy projects.
>

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Philip K.]

Zihao Zhu <cjpeople2013@gmail.com> writes:

> IMO, it's not practical to use directory local variable
>
> 1. directory local variable goes ".dir-local.el". But we can't mark 
> every directory contain this file as project. We have to do extra search 
> if we use directory local variable.

Not necessarily, if the directory variable would contain a (relative)
path to where the project root is.

> In your use case. I think we can add variable "project-known-projects", 
> you can  add your favorite directory to it, you can also persist this 
> variable using savehist.el

I actually think this is better (given there is a command to add a
directory to the list of known projects), because you could also build a
project switcher on this foundation.

And if the variable is a user option (which I think it should be),
savehist shouldn't be necessary.

-- 
	Philip K.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Basil L. Contovounesios]

"Philip K." <philip@warpmail.net> writes:

> Zihao Zhu <cjpeople2013@gmail.com> writes:
>
>> In your use case. I think we can add variable "project-known-projects", 
>> you can  add your favorite directory to it, you can also persist this 
>> variable using savehist.el
>
> I actually think this is better (given there is a command to add a
> directory to the list of known projects), because you could also build a
> project switcher on this foundation.
>
> And if the variable is a user option (which I think it should be),
> savehist shouldn't be necessary.

FWIW, Emacs 28 now has the user option project-list-file, which is a
file populated with the list of known projects maintained in the
variable project--list.  See the following emacs-devel discussion:

https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg03301.html
https://lists.gnu.org/archive/html/emacs-devel/2020-06/msg00035.html

-- 
Basil

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

IMO, it's not practical to use directory local variable

1. directory local variable goes ".dir-local.el". But we can't mark every
directory contain this file as project. We have to do extra search if we
use directory local variable.

2. If we have variable "project-directory-plain-project-p", It's a problem
for us to determine the root


On 2020/5/28 下午3:42, Philip K. wrote:

Zhu Zihao <cjpeople2013@gmail.com> <cjpeople2013@gmail.com> writes:


To mark a directoy as project, put an empty magic file .emacs-project under the
directory, and project.el should be responsible for it.

Is there any more standard name than ".emacs-project"? Or could a
directory local-variable be used? I like the idea, but wouldn't want to
have so many ".emacs-project" files lying around in toy projects.

[-- Attachment #2: Type: text/html, Size: 1232 bytes --]

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

Hi!

On 28.05.2020 06:32, Zhu Zihao wrote:
> This patch add support for "plain project" in project.el. Plain project is a
> kind of project without any VC backend but should be.
> 
> To mark a directoy as project, put an empty magic file .emacs-project 
> under the
> 
> directory, and project.el should be responsible for it.

Is that really a good idea?

I mean, you of course can set up a project type like that yourself.

But if it's included in project.el, it means we're taking it seriously. 
And there's no way to specify the ignored files, say. And file 
enumeration will inevitably be slower than in VC-based projects.

Do you have a lot of projects that aren't backed by VC repositories?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Zihao Zhu]

If I have a project.el based plugin,  and  I wanna use them in a 
directory not under VC. Add an mark to force project.el work is easier 
than modify the source of plugin or initialize VC system.

And this  can also be used as a side solution to use project.el in 
unsupported VCed project in Emacs (AFAIK, P4(Perforce) is not supported 
by vc.el).

IMO, a plain project is like a transient project.

On 2020/5/28 下午8:35, Dmitry Gutov wrote:
> Hi!
>
> On 28.05.2020 06:32, Zhu Zihao wrote:
>> This patch add support for "plain project" in project.el. Plain 
>> project is a
>> kind of project without any VC backend but should be.
>>
>> To mark a directoy as project, put an empty magic file .emacs-project 
>> under the
>>
>> directory, and project.el should be responsible for it.
>
> Is that really a good idea?
>
> I mean, you of course can set up a project type like that yourself.
>
> But if it's included in project.el, it means we're taking it 
> seriously. And there's no way to specify the ignored files, say. And 
> file enumeration will inevitably be slower than in VC-based projects.
>
> Do you have a lot of projects that aren't backed by VC repositories?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 28.05.2020 18:46, Zihao Zhu wrote:
> If I have a project.el based plugin,  and  I wanna use them in a 
> directory not under VC. Add an mark to force project.el work is easier 
> than modify the source of plugin or initialize VC system.

The problem with that, is that next time we'll get a report that these 
projects are too slow. Or that people who added .emacs-project file in 
the middle of a VC repository suddenly get significantly worse file 
listing performance, without expecting it.

So we'd have to add caching to the file list, and then some 
invalidation, probably. And I'm not a fan of having manual invalidation 
commands.

That's why I'm wary of adding such a separate project type by default, 
especially when the initial proposal doesn't add any of the advanced 
features described above, or explains how they won't be necessary.

But opinions welcome.

> And this  can also be used as a side solution to use project.el in 
> unsupported VCed project in Emacs (AFAIK, P4(Perforce) is not supported 
> by vc.el).

Perhaps we could add Perforce support to project-vc instead?

There was a vc-p4 packages somewhere out there. But if it's entirely 
dead, we could add such support to project--vc-list-files directly.

Or, better yet, release it as a project-p4 package on GNU ELPA.

That all depends, of course, on whether the p4 command line utility also 
has the means to quickly list repository files and add ignores.

> IMO, a plain project is like a transient project.

One difference is, nobody really expects much from "transient" projects. 
And this type of project is only applied when the directory is not 
covered by any other kind of project.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Zihao Zhu]

Besides Perforce, Darcs and Fossil aren't supported yet. If we have 
plain project, we don't have to fiddle around with different sorts of VC.

A more netrual suggestion: We can make project-find-functions a 
defcustom first.


On 2020/5/29 上午3:58, Dmitry Gutov wrote:
> On 28.05.2020 18:46, Zihao Zhu wrote:
>> If I have a project.el based plugin,  and I wanna use them in a 
>> directory not under VC. Add an mark to force project.el work is 
>> easier than modify the source of plugin or initialize VC system.
>
> The problem with that, is that next time we'll get a report that these 
> projects are too slow. Or that people who added .emacs-project file in 
> the middle of a VC repository suddenly get significantly worse file 
> listing performance, without expecting it.
>
> So we'd have to add caching to the file list, and then some 
> invalidation, probably. And I'm not a fan of having manual 
> invalidation commands.
>
> That's why I'm wary of adding such a separate project type by default, 
> especially when the initial proposal doesn't add any of the advanced 
> features described above, or explains how they won't be necessary.
>
> But opinions welcome.
>
>> And this  can also be used as a side solution to use project.el in 
>> unsupported VCed project in Emacs (AFAIK, P4(Perforce) is not 
>> supported by vc.el).
>
> Perhaps we could add Perforce support to project-vc instead?
>
> There was a vc-p4 packages somewhere out there. But if it's entirely 
> dead, we could add such support to project--vc-list-files directly.
>
> Or, better yet, release it as a project-p4 package on GNU ELPA.
>
> That all depends, of course, on whether the p4 command line utility 
> also has the means to quickly list repository files and add ignores.
>
>> IMO, a plain project is like a transient project.
>
> One difference is, nobody really expects much from "transient" 
> projects. And this type of project is only applied when the directory 
> is not covered by any other kind of project.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Philip K.]

Zihao Zhu <cjpeople2013@gmail.com> writes:

> Besides Perforce, Darcs and Fossil aren't supported yet. If we have
> plain project, we don't have to fiddle around with different sorts of
> VC.

Actually Fossil and Dards are supported, but you have to install the vc
backends:

- http://chiselapp.com/user/venks/repository/emacs-fossil
- https://www.irif.fr/~jch//software/repos/vc-darcs/

-- 
	Philip K.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 29.05.2020 07:34, Zihao Zhu wrote:
> 
> A more netrual suggestion: We can make project-find-functions a 
> defcustom first.

It's a hook. It's stable, you can use it.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Juri Linkov]

> Do you have a lot of projects that aren't backed by VC repositories?

Usually new projects in early stages of development
not ready to commit even the initial revision.

However, maybe in this case better to employ some heuristics
to detect the project root, e.g. to search for such common files
as "Gemfile" for Ruby, "mix.exs" for Elixir, etc.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 03.06.2020 02:40, Juri Linkov wrote:
>> Do you have a lot of projects that aren't backed by VC repositories?
> 
> Usually new projects in early stages of development
> not ready to commit even the initial revision.

A 'git init' wouldn't exactly hurt, though.

> However, maybe in this case better to employ some heuristics
> to detect the project root, e.g. to search for such common files
> as "Gemfile" for Ruby, "mix.exs" for Elixir, etc.

I like this suggestion better (no "special" files), but would we be able 
to avoid scope creep? Wouldn't users then expect being able to specify 
ignored directories in such projects? And then faster indexing (e.g. 
using caching), compared to the VC backend?

Projectile uses the "special file" for the former, and I'm not really 
fond of its solution for the latter.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Theodor Thornhill]

       
Hello,       
       
"Dmitry Gutov" <dgutov@yandex.ru> writes:

> I like this suggestion better (no "special" files), but would we be able
> to avoid scope creep? Wouldn't users then expect being able to specify
> ignored directories in such projects? And then faster indexing (e.g.
> using caching), compared to the VC backend?

Isn't this already supported?
In a major mode I'm making right now I believe I have covered both options:

(defcustom elm-root-file "elm.json"
  "...")   
   
(defun elm-project-root (dir)
  "Create the cons cell `project-root' needs to discover root."
  (let ((root (locate-dominating-file dir elm-root-file)))
    (when root
      (cons 'elm root))))
  
(cl-defmethod project-root ((project (head elm)))
  (cdr project))

(cl-defmethod project-ignores ((project (head elm)) dir)
  (append vc-directory-exclusion-list
          ;; This could also be a defcustom ofc
          (list "./elm-stuff/")))

(add-hook 'project-find-functions #'elm-project-root)

- This will add both the option to not execute git init, but still find root.
- Also grepping will ignore the "node-modulesy" "elm-stuff".
- In addition, the vc-directory-exclusion-list can be edited to support arbitrary patterns, at least with my limited testing.
- It should be no problem to add a ".emacs-project" there in your own config?
  
Am I missing something, or is this already available? This of course depends on major modes supporting this integration?
  
Theodor  
  
>
> Projectile uses the "special file" for the former, and I'm not really
> fond of its solution for the latter.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

Hi!

On 05.06.2020 22:22, Theodor Thornhill wrote:
> "Dmitry Gutov" <dgutov@yandex.ru> writes:
> 
>> I like this suggestion better (no "special" files), but would we be able
>> to avoid scope creep? Wouldn't users then expect being able to specify
>> ignored directories in such projects? And then faster indexing (e.g.
>> using caching), compared to the VC backend?
> 
> Isn't this already supported?

Not in the "plain" project backend as proposed.

> In a major mode I'm making right now I believe I have covered both options:

Ignoring is covered by the API, yes. How did you cover the caching issue?

Also note that unless your new project backend is "good enough", you 
might make the users' experience worse as a result, at least in some 
respects.

> (defcustom elm-root-file "elm.json"
>    "...")
>     
> (defun elm-project-root (dir)
>    "Create the cons cell `project-root' needs to discover root."
>    (let ((root (locate-dominating-file dir elm-root-file)))
>      (when root
>        (cons 'elm root))))
>    
> (cl-defmethod project-root ((project (head elm)))
>    (cdr project))
> 
> (cl-defmethod project-ignores ((project (head elm)) dir)
>    (append vc-directory-exclusion-list
>            ;; This could also be a defcustom ofc
>            (list "./elm-stuff/")))
> 
> (add-hook 'project-find-functions #'elm-project-root)

The code is good. With probably one exception: if you have a big enough 
project, and it's checked into Git, the VC project will be much faster 
at indexing files than your project implementation (because 
project-files by default uses 'find').

But if you also define a faster project-files override, it should be fine.

Another issue is, well, if the user has a different package that defines 
projects installed (maybe Projectile grows project.el integration 
someday), or they're simply used to the VC backend, they might be 
surprised with some finer details unless you clearly document that your 
package does add a new project backend.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Theodor Thornhill]

       
"Dmitry Gutov" <dgutov@yandex.ru> writes:

Hi!

> "Dmitry Gutov" <dgutov@yandex.ru> writes:

[...]
> Not in the "plain" project backend as proposed.

I see your point.
  
[...]

> Another issue is, well, if the user has a different package that defines
> projects installed (maybe Projectile grows project.el integration
> someday), or they're simply used to the VC backend, they might be
> surprised with some finer details unless you clearly document that your
> package does add a new project backend.

Thanks - Wasn't aware of this. Seems like a better solution over all is to enforce the vc-backend? It seems like we get the "transient" version, or the "vc" version, but defining your own will have several drawbacks?

Theo

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 06.06.2020 11:48, Theodor Thornhill wrote:
> Thanks - Wasn't aware of this. Seems like a better solution over all is to enforce the vc-backend? It seems like we get the "transient" version, or the "vc" version, but defining your own will have several drawbacks?

Unless you make sure it's full-featured, indeed. But the problem might 
become more severe in the future if we add more capabilities to projects.

This particular problem with speed could be alleviated if we export a 
utility function similar to project--vc-list-files, so that other impls 
could use Git's file listing speed.

The primary drawback is the semantics: the current impl always follows 
.gitignore for its ignores (but accepts additional ones), whereas an 
arbitrary project can have a totally different set of ignores. So, at 
the very least, I'm in doubt how to write the docstring.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On Sat, Jun 6, 2020 at 13:15, Dmitry Gutov <dgutov@yandex.ru> wrote:

>> Unless you make sure it's full-featured, indeed. But the problem might
> become more severe in the future if we add more capabilities to projects.

This would essentially mean that at least until the api is completely stable the git version is the de facto implementation?

How about as you proposed extract the vc until, then compose different versions? Most projects will use it anyways, and my foo-mode shouldn’t have miss out.

I mean, if we want to support “.project”, I assume we still want to use vc backend after we do git init. Should we have to delete that said file then? What if we accept some pattern, then merge all the other functions? I believe we can’t use generics, call-next-method and friends for this?

>> arbitrary project can have a totally different set of ignores. So, at
> the very least, I'm in doubt how to write the docstring.

An arbitrary project can then just add a “list file“ functions, and if git is not there, it will just return nil

Theo

[-- Attachment #2: Type: text/html, Size: 1491 bytes --]

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 06.06.2020 14:53, Theodor Thornhill wrote:
> 
> 
> On Sat, Jun 6, 2020 at 13:15, Dmitry Gutov <dgutov@yandex.ru 
> <mailto:dgutov@yandex.ru>> wrote:
>> > Unless you make sure it's full-featured, indeed. But the problem might
>> become more severe in the future if we add more capabilities to projects.
> This would essentially mean that at least until the api is completely 
> stable the git version is the de facto implementation?

It's the "official" one, but, for instance, Projectile which we already 
mentioned before, could hook into the API and provide the same level of 
performance.

As far as future features go, it could be a concern, but can be 
addressed as the features appear.

The crucial point is, though, that "lightweight" project backends are 
probably not a good idea.

> How about as you proposed extract the vc until, then compose different 
> versions? Most projects will use it anyways, and my foo-mode shouldn’t 
> have miss out.

Not sure what you're saying here.

> I mean, if we want to support “.project”, I assume we still want to use 
> vc backend after we do git init. Should we have to delete that said file 
> then?

I think the "simple" backend, if added, would go after the VC one in 
project-find-functions. So doing 'git init' would automatically switch 
over, with all the accompanying consequences.

> What if we accept some pattern, then merge all the other 
> functions? I believe we can’t use generics, call-next-method and friends 
> for this?

Could you elaborate?

>> > arbitrary project can have a totally different set of ignores. So, at
>> the very least, I'm in doubt how to write the docstring.
>>
> An arbitrary project can then just add a “list file“ functions, and if 
> git is not there, it will just return nil

How does that address the question of ignores?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 796 bytes --]

> Could you elaborate?

Actually, I think I’ll withdraw the over engineering tendency of mine here and rather refine:

I believe it is appropriate to view project.el to have two different focuses here: one for the general idea and one minor mode (overloaded term, I know...) for vc.

To help the out of the box experience with the project-vc I think either:

- adding common patterns to project-vc-ignores instead of nil, to accommodate for improperly configured .gitignore

- simply adding documentation emphasizing project-vcs reliance on .gitignore

As such it could be neither project nor project-vc responsibility to add the plain version.

Will project.el be split at some point? I mean moving vc things to vc.
I for one (maybe only one) find the scope of project.el a bit confusing

Theo

[-- Attachment #2: Type: text/html, Size: 1160 bytes --]

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 06.06.2020 16:37, Theodor Thornhill wrote:
> 
> 
>> Could you elaborate?
> 
> Actually, I think I’ll withdraw the over engineering tendency of mine 
> here and rather refine:
> 
> I believe it is appropriate to view project.el to have two different 
> focuses here: one for the general idea and one minor mode (overloaded 
> term, I know...) for vc.

Right.

> To help the out of the box experience with the project-vc I think either:
> 
> - adding common patterns to project-vc-ignores instead of nil, to 
> accommodate for improperly configured .gitignore
> 
> - simply adding documentation emphasizing project-vcs reliance on .gitignore

I have hopefully addressed the above items in commit 7259377. Let me 
know if you think the description is missing something.

> As such it could be neither project nor project-vc responsibility to add 
> the plain version.

project-vc could morph into something like "project-auto"... with a 
customizable list of project markers.

Still thinking about it.

> Will project.el be split at some point? I mean moving vc things to vc.
> I for one (maybe only one) find the scope of project.el a bit confusing

That would be a good idea, but I think that would make it two different 
packages. ELPA Core doesn't understand multifile packages, AFAIK.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Lars Ingebrigtsen]

If I'm skimming this thread correctly, there wasn't much enthusiasm for
the proposed new functionality, so I'm closing this bug report.  If
there's further progress to be made here, please respond to the debbugs
address and we'll reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

On 01.10.2020 06:11, Lars Ingebrigtsen wrote:
> If I'm skimming this thread correctly, there wasn't much enthusiasm for
> the proposed new functionality, so I'm closing this bug report.  If
> there's further progress to be made here, please respond to the debbugs
> address and we'll reopen.

There are valid issues here, but there was no clear picture of how to 
solve them best. But it would be good to try to do that (at least to 
some extent) before Emacs 28 is released.

The first issue is that projects that don't use any of the supported VCS 
are not recognized at all.

Since these days most large projects do use version control, we can 
expect that projects which do not, should be relatively small. And our 
find-based implementation will generally be fast enough. Also, looking 
at the home-grown project definitions out there, people often don't 
bother with the 'ignores' method, so maybe a backend which provides no 
way to specify them will still cover most of the needs. It could be 
extended with such feature later anyway.

Patch adding such backend with a customizable list of "markers" is 
attached. I wasn't sure whether to add support for wildcards there, 
because file-expand-wildcards does not optimize for the trivial case, 
and directory-files is slower than file-exists-p. I'm mostly worried 
about Tramp performance here, but some pathologically huge directory 
could be a problem as well, IDK.

project-fallback-markers only includes .dir-locals.el, but it can be 
easily customized. Not sure if it should include other standard file 
names, such as Makefile, Gemfile, mix.exs, and so on. The first version 
doesn't have them.

Another question is the name. 'plain' sounds not very descriptive. I've 
considered 'fallback' and 'novc', eventually settling on the former.

Opinions welcome, though.

[-- Attachment #2: project-fallback.diff --]
[-- Type: text/x-patch, Size: 1371 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 57a961c260..8eee0c7d27 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -165,7 +165,7 @@ project
   :version "28.1"
   :group 'tools)
 
-(defvar project-find-functions (list #'project-try-vc)
+(defvar project-find-functions (list #'project-try-vc #'project-try-fallback)
   "Special hook to find the project containing a given directory.
 Each functions on this hook is called in turn with one
 argument, the directory in which to look, and should return
@@ -662,6 +662,27 @@ project-buffers
         (push buf bufs)))
     (nreverse bufs)))
 
+(defcustom project-fallback-markers '(".dir-locals.el")
+  "List of file name to use as fallback project root markers.
+
+Each entry should be a relative file name (no directory part).
+
+The fallback project backend detects a project as a directory
+containing one of these files."
+  :type 'list)
+
+(defun project-try-fallback (dir)
+  (let ((root (locate-dominating-file dir #'project-fallback--root-p)))
+    (and root (cons 'fallback root))))
+
+(defun project-fallback--root-p (dir)
+  (let ((default-directory dir))
+    (seq-some (lambda (marker) (file-exists-p marker))
+              project-fallback-markers)))
+
+(cl-defmethod project-root ((project (head fallback)))
+  (cdr project))
+
 \f
 ;;; Project commands
 

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> There are valid issues here, but there was no clear picture of how to
> solve them best. But it would be good to try to do that (at least to
> some extent) before Emacs 28 is released.

OK; I'm reopening this report, then.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Juri Linkov]

> Patch adding such backend with a customizable list of "markers" is
> attached.

I've tested that it allows using 'C-x p g' in any directory,
not only in a repository!  This is a great feature, thanks.

But a small problem that there is a need to exclude some subdirectories
from the search by 'C-x p g'.  And I don't understand if your another patch
with 'project-vc-subprojects' could help with this?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 03.10.2021 21:06, Juri Linkov wrote:
>> Patch adding such backend with a customizable list of "markers" is
>> attached.
> 
> I've tested that it allows using 'C-x p g' in any directory,
> not only in a repository!  This is a great feature, thanks.

Thanks for testing.

> But a small problem that there is a need to exclude some subdirectories
> from the search by 'C-x p g'.  And I don't understand if your another patch
> with 'project-vc-subprojects' could help with this?

Now, it's for something else (subprojects within monorepos).

A way to specify ignore entries for "fallback" projects can be added 
easily with a variable like project-fallback-ignores (same as 
project-vc-ignores). I've been debating internally whether we need such 
a variable at all, and whether it should simply be an alias for 
project-vc-ignores. But, uh, probably not?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 05.10.2021 05:03, Dmitry Gutov wrote:
> Now, it's for something else (subprojects within monorepos).

s/Now/No

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Juri Linkov]

> A way to specify ignore entries for "fallback" projects can be added easily
> with a variable like project-fallback-ignores (same as
> project-vc-ignores). I've been debating internally whether we need such
> a variable at all, and whether it should simply be an alias for
> project-vc-ignores. But, uh, probably not?

Adding such a line to .dir-locals.el would be fine:

  ((nil . ((project-vc-ignores . '("subdir1/" "subdir2/")))))

Or this would be even better:

  ((nil . ((project-ignores . '("subdir1/" "subdir2/")))))

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 05.10.2021 09:52, Juri Linkov wrote:
>> A way to specify ignore entries for "fallback" projects can be added easily
>> with a variable like project-fallback-ignores (same as
>> project-vc-ignores). I've been debating internally whether we need such
>> a variable at all, and whether it should simply be an alias for
>> project-vc-ignores. But, uh, probably not?
> Adding such a line to .dir-locals.el would be fine:
> 
>    ((nil . ((project-vc-ignores . '("subdir1/" "subdir2/")))))

     ((nil . ((project-fallback-ignores . '("subdir1/" "subdir2/")))))

> Or this would be even better:
> 
>    ((nil . ((project-ignores . '("subdir1/" "subdir2/")))))

I wouldn't want to create an impression that this variable affects any 
and all project.el backends, including third-party ones.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Juri Linkov]

>>> A way to specify ignore entries for "fallback" projects can be added easily
>>> with a variable like project-fallback-ignores (same as
>>> project-vc-ignores). I've been debating internally whether we need such
>>> a variable at all, and whether it should simply be an alias for
>>> project-vc-ignores. But, uh, probably not?
>> Adding such a line to .dir-locals.el would be fine:
>>    ((nil . ((project-vc-ignores . '("subdir1/" "subdir2/")))))
>
>     ((nil . ((project-fallback-ignores . '("subdir1/" "subdir2/")))))

This is fine too.

>> Or this would be even better:
>>    ((nil . ((project-ignores . '("subdir1/" "subdir2/")))))
>
> I wouldn't want to create an impression that this variable affects any and
> all project.el backends, including third-party ones.

Ok.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Juri Linkov]

>>>> A way to specify ignore entries for "fallback" projects can be added easily
>>>> with a variable like project-fallback-ignores (same as
>>>> project-vc-ignores). I've been debating internally whether we need such
>>>> a variable at all, and whether it should simply be an alias for
>>>> project-vc-ignores. But, uh, probably not?
>>> Adding such a line to .dir-locals.el would be fine:
>>>    ((nil . ((project-vc-ignores . '("subdir1/" "subdir2/")))))
>>
>>     ((nil . ((project-fallback-ignores . '("subdir1/" "subdir2/")))))
>
> This is fine too.

Actually, the name 'project-fallback-ignores' would be too weird
when used in .dir-locals.el.  The users will ask "Fall back from where?"

Maybe a better name would be 'project-directory-ignores'
with the directory-based backend name 'project-directory'?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Juri Linkov]

>>>     ((nil . ((project-fallback-ignores . '("subdir1/" "subdir2/")))))
>>
>> This is fine too.
>
> Actually, the name 'project-fallback-ignores' would be too weird
> when used in .dir-locals.el.  The users will ask "Fall back from where?"
>
> Maybe a better name would be 'project-directory-ignores'
> with the directory-based backend name 'project-directory'?

Or simply 'project-dir-ignores' where 'dir' refers to 'dir-locals'.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 06.10.2021 19:29, Juri Linkov wrote:
>>>>      ((nil . ((project-fallback-ignores . '("subdir1/" "subdir2/")))))
>>>
>>> This is fine too.
>>
>> Actually, the name 'project-fallback-ignores' would be too weird
>> when used in .dir-locals.el.  The users will ask "Fall back from where?"
>>
>> Maybe a better name would be 'project-directory-ignores'
>> with the directory-based backend name 'project-directory'?
> 
> Or simply 'project-dir-ignores' where 'dir' refers to 'dir-locals'.

Wouldn't that imply that it would apply to every backend, as long as the 
entry is in .dir-locals.el?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 06.10.2021 10:21, Juri Linkov wrote:

> Actually, the name 'project-fallback-ignores' would be too weird
> when used in .dir-locals.el.  The users will ask "Fall back from where?"

It's the standard practice: using the prefix, so it's "ignores 
pertaining to the project-fallback project backend".

Fallback from what? From the VC backend. The fallback backend is used 
when the VC backend is not (there is no recognized VC repository).

> Maybe a better name would be 'project-directory-ignores'
> with the directory-based backend name 'project-directory'?

I don't know if it's better. What does "directory" mean? Every backend, 
every project has directories.

As mentioned previously, the other option I had considered was 'novc'. 
Then the variable would be called project-novc-ignores.

This is not a done deal, just what seems the most optimal to me at the 
moment. But opinions welcome.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Juri Linkov]

>> Maybe a better name would be 'project-directory-ignores'
>> with the directory-based backend name 'project-directory'?
>
> I don't know if it's better. What does "directory" mean? Every backend,
> every project has directories.

Then maybe the backend could be named 'project-file'
since a special file defines the project root.

> As mentioned previously, the other option I had considered was 'novc'. Then
> the variable would be called project-novc-ignores.

"novc" is the worst variant.  It's not obvious that it means
"no-version-control", and also will make no sense when more backends
will be added.  Or no more backends are planned, and all other possible
roots should be handled by the same fallback backend?  Or would it be
possible that more backends could be added in project-find-functions
after the file-based fallback backend?  Then the name "fallback"
will make no sense if it will not be the last in project-find-functions.

> This is not a done deal, just what seems the most optimal to me at the
> moment. But opinions welcome.

Maybe it will help to choose a better name while thinking about
more possible backends that could be added later.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 07.10.2021 10:17, Juri Linkov wrote:
>>> Maybe a better name would be 'project-directory-ignores'
>>> with the directory-based backend name 'project-directory'?
>>
>> I don't know if it's better. What does "directory" mean? Every backend,
>> every project has directories.
> 
> Then maybe the backend could be named 'project-file'
> since a special file defines the project root.

That's a little more meaningful, though too close to 'project-files'. 
'project-markered' or 'project-markerfile' would probably be less 
ambiguous. But... (*)

>> As mentioned previously, the other option I had considered was 'novc'. Then
>> the variable would be called project-novc-ignores.
> 
> "novc" is the worst variant.  It's not obvious that it means
> "no-version-control", and also will make no sense when more backends
> will be added.

Might not be obvious, but when you know what 'vc' stands for, 'novc' 
should at least be a strong hint. And then you could reach for 
documentation.

But indeed it's very short and might make less sense if more backends 
are configured.

> Or no more backends are planned, and all other possible
> roots should be handled by the same fallback backend?  Or would it be
> possible that more backends could be added in project-find-functions
> after the file-based fallback backend?  Then the name "fallback"
> will make no sense if it will not be the last in project-find-functions.

I don't know about the case of adding more backends at the end of 
project-find-functions (are there any people who have done this?), but 
otherwise, 'fallback' is both an indication that the backend is at the 
end, and a strong hint to avoid moving it to the front.

Suppose somebody puts it before 'vc' to use if for a purpose we did not 
design it for: make sure that some subproject 'foo' in their monorepo is 
considered a separate project. 'foo/Makefile' exists, so they add 
"Makefile" to project-fallback-markers, and it kind of seems to work.

But! File listing performance becomes worse: we didn't optimize this 
backend for use inside of (e.g.) Git repositories, we don't take 
advantage of 'git ls-files'.

More than that, it doesn't honor the ignore instructions from 
.gitignore. Whereas, in a lot of cases, it would be helpful (and even 
necessary for good performance) to honor them. But on the other hand, 
why would a 'project-fallback', or 'project-file', or 
'project-markerfile', honor .gitignore contents? Semantically, that 
doesn't make much sense. And yet, I'd wager like a half of the users 
would implicitly expect it to do so, and another half -- would not. 
That's doesn't seem like a great design.

>> This is not a done deal, just what seems the most optimal to me at the
>> moment. But opinions welcome.
> 
> Maybe it will help to choose a better name while thinking about
> more possible backends that could be added later.

(*) ...it doesn't seem compatible with being used in arbitrary order 
with arbitrary backends.

Perhaps, if we change our mind on the overall design later, we could 
create a new backend, with a different name and more complex 
implementation logic.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Juri Linkov]

>> Then maybe the backend could be named 'project-file'
>> since a special file defines the project root.
>
> That's a little more meaningful, though too close to
> 'project-files'. 'project-markered' or 'project-markerfile' would probably
> be less ambiguous.

In 'project-filemarker' I misread "filemarker" as "filmmaker" :-)
Another possible name would be "fileroot".

> Suppose somebody puts it before 'vc' to use if for a purpose we did not
> design it for: make sure that some subproject 'foo' in their monorepo is
> considered a separate project. 'foo/Makefile' exists, so they add
> "Makefile" to project-fallback-markers, and it kind of seems to work.

There are two contradictory needs:

1. When a marker list contains both ".dir-locals.el" and "Makefile",
   it should ignore Makefile files in vc-based project subdirs, e.g.
   emacs/lisp/Makefile, etc.

2. OTOH, I often type 'C-x p g' to search all gems of the same
   ruby version in e.g. ~/.rbenv/versions/2.7.4/lib/ruby/gems
   But it finds ~/.rbenv/.git and tries to search all ruby versions.
   I could manually add .dir-locals.el only to a particular version's
   subdir.  But how to override ~/.rbenv/.git?  Maybe by changing
   the order of backends in project-find-functions?
   Then the fallback won't be the last backend anymore.
   Also the backend priorities will be changed globally
   for all other projects, and 'C-x p g' in emacs/lisp
   will find emacs/lisp/Makefile to override emacs/.git.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 10.10.2021 19:47, Juri Linkov wrote:
>>> Then maybe the backend could be named 'project-file'
>>> since a special file defines the project root.
>>
>> That's a little more meaningful, though too close to
>> 'project-files'. 'project-markered' or 'project-markerfile' would probably
>> be less ambiguous.
> 
> In 'project-filemarker' I misread "filemarker" as "filmmaker" :-)

Right. :-)

> Another possible name would be "fileroot".

Also sounds more like a method than a project type name.

Maybe project-markered or project-marked, or project-dominated (along 
the lines of 'locate-dominating-file'?). Or something noncommittal like 
project-dirtee.

But all of those sounds like one could put them at any position in 
project-find-functions, which project-fallback explicitly discourages.

>> Suppose somebody puts it before 'vc' to use if for a purpose we did not
>> design it for: make sure that some subproject 'foo' in their monorepo is
>> considered a separate project. 'foo/Makefile' exists, so they add
>> "Makefile" to project-fallback-markers, and it kind of seems to work.
> 
> There are two contradictory needs:
> 
> 1. When a marker list contains both ".dir-locals.el" and "Makefile",
>     it should ignore Makefile files in vc-based project subdirs, e.g.
>     emacs/lisp/Makefile, etc.

Right. That says project-try-fallback going after project-try-vc is a 
good thing.

> 2. OTOH, I often type 'C-x p g' to search all gems of the same
>     ruby version in e.g. ~/.rbenv/versions/2.7.4/lib/ruby/gems
>     But it finds ~/.rbenv/.git and tries to search all ruby versions.
>     I could manually add .dir-locals.el only to a particular version's
>     subdir.

I'm using this setup, FWIW:

   (defun project-try-gem (dir)
     (when (string-match "/\\.rbenv/versions/.*/gems/.*/gems/[^/]+/" dir)
       (cons 'rubygem
             (substring dir 0 (match-end 0)))))

   (cl-defmethod project-root ((project (head rubygem)))
     (cdr project))

   (with-eval-after-load 'project
     (add-hook 'project-find-functions #'project-try-gem))

Which avoids the whole directory-walking routine, probably saving on a 
number of CPU cycles.

> But how to override ~/.rbenv/.git?  Maybe by changing
>     the order of backends in project-find-functions?
>     Then the fallback won't be the last backend anymore.
>     Also the backend priorities will be changed globally
>     for all other projects, and 'C-x p g' in emacs/lisp
>     will find emacs/lisp/Makefile to override emacs/.git.

If we keep the same tool to ensure the priorities (order of functions in 
project-try-vc), that would seem like we should use two different 
backends for these two different purposes.

Say, the first one would be called project-dominating, and the other one 
- still project-fallback.

project-find-functions will be

   '(project-try-dominating project-try-vc project-try-fallback

Alternatively, one backend could combine these, but it would have, like, 
configurable logic with two different sets of predicates -- one 
universal (whether the file is inside a VCS repo or not), and another 
for when the file is strictly outside of VCS repos. But that sounds 
trickier, both to configure and to understand. Also, it seems like with 
this approach the backend *should* ignore the .gitignore entires, which 
is fine for .rbenv/versions/.*/gems, but it's bound not to work for some 
other purposes some users are going to try to use it for.

So, since I don't know of any other use cases for the project-dominating 
code path rather than Ruby gems inside ~/.rbenv, I figured not to try to 
solve this use case yet. But we can catalogue similar use cases (maybe 
other versions switchers, for Ruby and other languages?; not sure if 
installed libraries for other languages also fit this approach) and add 
another backend for them later. Help welcome, of course.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Nikolay Kudryavtsev]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

Hello. I have some stake in this issue too.

The proposed patch is ok, but I think that we can go for a more 
ambitious solution here that would require just a little more work, but 
would be a lot more useful.

Maybe lets have a make-file-backend function, the output of which you 
can cons onto project-find-functions, so any file defined like this 
would be just another separate backend.

This function in turn can be used by major mode developers and users to 
implement project backends. If we look at the list of projects supported 
by Projectile 
<https://docs.projectile.mx/projectile/projects.html#file-markers>, we 
can see that this one function is enough already to implement most of 
them. Having such file backends also leaves the question of whether the 
particular file should have a higher priority than VC to the end user, 
depending on where he(or some major mode he's using) adds a particular 
one into the list.

As for whether there should be some kind of default file marked project, 
sure, the user can just remove it if he want to.


[-- Attachment #2: Type: text/html, Size: 1364 bytes --]

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

Hi Nikolay,

On 05.10.2021 17:39, Nikolay Kudryavtsev wrote:
> Hello. I have some stake in this issue too.
> 
> The proposed patch is ok, but I think that we can go for a more 
> ambitious solution here that would require just a little more work, but 
> would be a lot more useful.
> 
> Maybe lets have a make-file-backend function, the output of which you 
> can cons onto project-find-functions, so any file defined like this 
> would be just another separate backend.
> 
> This function in turn can be used by major mode developers and users to 
> implement project backends. If we look at the list of projects supported 
> by Projectile 
> <https://docs.projectile.mx/projectile/projects.html#file-markers>, we 
> can see that this one function is enough already to implement most of 
> them. Having such file backends also leaves the question of whether the 
> particular file should have a higher priority than VC to the end user, 
> depending on where he(or some major mode he's using) adds a particular 
> one into the list.

Is this pretty much to be able to choose the backends order?

I'm afraid it's not a good idea: any backend that comes before the VC 
backend can make the user experience worse. Because Git-based (fast) 
file listing is only implemented for the VC backend.

> As for whether there should be some kind of default file marked project, 
> sure, the user can just remove it if he want to.

And you can customize which file names serve as markers.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Nikolay Kudryavtsev]

It's mostly to somewhat empower people to add new backends.

But being to chose backend order is helpful too, since lets say I'm a 
major mode developer. I add my project type to the end of the list. Then 
someone says "hey, my VC structure is non-standard so the project gets 
detected wrongly" and I can just point the user to reorder his list.

I don't think we should worry about the user experience getting worse 
too much, since anyone touching project-find-functions should know what 
they're doing.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 05.10.2021 18:21, Nikolay Kudryavtsev wrote:
> It's mostly to somewhat empower people to add new backends.

It's not too difficult to do even now.

> But being to chose backend order is helpful too, since lets say I'm a 
> major mode developer. I add my project type to the end of the list. Then 
> someone says "hey, my VC structure is non-standard so the project gets 
> detected wrongly" and I can just point the user to reorder his list.

As long as major mode developers don't do that, it's probably fine.

But you can provide a backend of your own

> I don't think we should worry about the user experience getting worse 
> too much, since anyone touching project-find-functions should know what 
> they're doing.

I wouldn't be so sure. Especially if you tell your users to do that.

But let's talk about "your project type". Is it an existing backend? 
What kind of projects are we talking about? Does it optimize the file 
listing performance?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Nikolay Kudryavtsev]

I currently have a very basic real use case on my hands. There's a 
particular programming language that has it's own project file type. 
Since it's a project type, it makes sense to plug it in as project 
backend. And then on top of this I can implement project target actions 
like build, compile and debug(this is actually another matter that 
probably needs to be implemented in(or over) project.el at some point, 
I'll probably open a discussion about it later).

So it all makes sense so far, at least conceptually. But here you're 
saying "you should not add a new file-based backend until you really 
think about the project file list optimization first". This violates the 
classic rule of doing the right thing first, then optimizing second. So 
while I feel this a real issue, it IMHO should be filed and discussed 
separately and is a nonblocker for this particular task.

Now to contradict myself, lets continue discussing this issue. I think 
this is a local version of a more global multiple backends problem. Lets 
say we have the same project(more precisely, a set of files) that is 
served by multiple backends. Roughly we order project-find-functions in 
this order: major-mode backends, tool backends(eg. GNU Global), generic 
backends(VC). The preference for the major mode backends over others is 
due to that "VC has different root" use case. Tool backends are 
preferred to VC due to that you can start a new Global project as sort 
of a custom project hack.

And here we run into the problem: our major mode while it provides a 
backend, does not optimize file listing, but there's a backend that 
does. I think TRT is project.el choosing a different secondary backend 
in this case as long as it has the same project root. For this it needs 
to have some rules, to know which backend better fits a particular 
situation, which does not.

This would require a change of backend registration, since you can't 
just have one function, so in the end the api would look something like 
this:

(register-project #'my-project-find-function :optimizes-file-listing nil)

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 05.10.2021 21:19, Nikolay Kudryavtsev wrote:
> I currently have a very basic real use case on my hands. There's a 
> particular programming language that has it's own project file type. 
> Since it's a project type, it makes sense to plug it in as project 
> backend. And then on top of this I can implement project target actions 
> like build, compile and debug(this is actually another matter that 
> probably needs to be implemented in(or over) project.el at some point, 
> I'll probably open a discussion about it later).

Right, we are yet to consider this functionality properly.

> So it all makes sense so far, at least conceptually. But here you're 
> saying "you should not add a new file-based backend until you really 
> think about the project file list optimization first". This violates the 
> classic rule of doing the right thing first, then optimizing second. So 
> while I feel this a real issue, it IMHO should be filed and discussed 
> separately and is a nonblocker for this particular task.

Let's talk about correctness. The "right thing".

Suppose we have a large Git repo, which contains a "foo project" (as you 
might see it), marked by Foofile in its 'foo' subdirectory.

And suppose we allow the project-foo backend to come first before the VC 
backend. When we are inside the directory ./project/foo, that's the 
current project. File listing shows ("./project/foo/a", 
"./project/foo/b", etc).

But when we go up a directory, "./project/". And when asked to list its 
files, how do we avoid including "./project/foo/a" in that list? It 
would make sense to exclude any nested projects, right? But we can't do 
that if the project detection logic is so flexible that the project-vc 
backend couldn't find out about subprojects inside it except by visiting 
every subdirectory and querying for the current project there, which 
would obviously be too slow.

So it's a correctness issue as well. Hence the 
simpler-but-easier-to-get-right approach in the other patch (with the 
project-vc-subprojects variable). Even if it might require more effort 
from the end user, unfortunately.

> Now to contradict myself, lets continue discussing this issue. I think 
> this is a local version of a more global multiple backends problem. Lets 
> say we have the same project(more precisely, a set of files) that is 
> served by multiple backends. Roughly we order project-find-functions in 
> this order: major-mode backends, tool backends(eg. GNU Global), generic 
> backends(VC). The preference for the major mode backends over others is 
> due to that "VC has different root" use case. Tool backends are 
> preferred to VC due to that you can start a new Global project as sort 
> of a custom project hack.

Setting project-find-functions in a major mode is a questionable thing 
to do, because then you end up with Emacs where files in the same 
directory belong to different projects. As we say in the commentary:

;; It is a good idea to depend on the
;; directory only, and not on the current major mode, for example.
;; Because the usual expectation is that all files in the directory
;; belong to the same project (even if some/most of them are ignored).

We want to support even backends where this approach is violated (on a 
best-effort basis), but let's not make this the common scenario.

> And here we run into the problem: our major mode while it provides a 
> backend, does not optimize file listing, but there's a backend that 
> does. I think TRT is project.el choosing a different secondary backend 
> in this case as long as it has the same project root.

If there is a next backend which indicates the same root, why do we need 
the first one?

> For this it needs 
> to have some rules, to know which backend better fits a particular 
> situation, which does not.

Why do you need so many backends, anyway? One per language, one per 
tool, etc. That seems to reduce the concept of a project to "this one 
parent directory containing a file some tool cares about".

What would be the meaning of the value (project-current) returns? 
Suppose I call project-find-file, meaning to jump to another file in the 
same Git repository. And instead I am shown only a list of files in the 
current subdirectory because it contains, say, a Makefile. Is that a 
good idea to enact this kind of behavior automatically?

Or suppose we add a backend that looks for 'Makefile', another for 
'Gemfile', another for 'Rakefile', etc. What user-level commands are 
going to benefit from this setup? A command that shows the available 
Makefile tasks? It can just as well call 'locate-dominating-file' to 
find the nearest directory containing it. Same for 'M-x rake', and so on.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Nikolay Kudryavtsev]

> But when we go up a directory, "./project/". And when asked to list 
> its files, how do we avoid including "./project/foo/a" in that list? 
> It would make sense to exclude any nested projects, right?
Not necessarily. It may be a useful thing to do for some projects, but 
it does not follow from anything that it should be the only or the 
default option. The correct solution here is IMHO implementing some kind 
of .project-settings.el file in which you can set hide-nested-project-files.

> Setting project-find-functions in a major mode is a questionable thing 
> to do, because then you end up with Emacs where files in the same 
> directory belong to different projects.
I'm not talking about setting it locally in the mode, more about modes 
providing such functions on load. That's another important question. 
More backends are more functions to test, so it's reasonable to add 
backends only when they're needed. I may avoid programming in language X 
from some months so no reason to keep that backed on, but when I start 
editing a file in that language, the major mode loads and so should the 
backend.

> If there is a next backend which indicates the same root, why do we 
> need the first one? 
We don't know what the next backend in line indicates. Nor do we care 
about it since the current one already gives us something. We just try 
to find backends until we find one in the order of priority and then stop.

> Suppose I call project-find-file, meaning to jump to another file in 
> the same Git repository. And instead I am shown only a list of files 
> in the current subdirectory because it contains, say, a Makefile. Is 
> that a good idea to enact this kind of behavior automatically? 
If a user believes that it looks like a duck, it should squeak like one. 
Sure you personally may want to suppress some possible project backends 
from firing, but someone may want the opposite.

I want to remind you of this recent-ish thread on HGE:

https://lists.gnu.org/archive/html/help-gnu-emacs/2021-09/msg00045.html

Lets take the maven example presented there. We have a project 
containing modules. The user wants to compile both independently. We can 
write two different compilation commands, one that works on the project 
and one that works on the module. Or we can just have a single command, 
since the compilation process is not different in any way for both. Then 
we can give that command some prefix that would make it work not on the 
project itself but on the root project of that project.

The Makefile example is your strongest argument here, but we can define 
find functions for it recursively. That is until you find a Makefile 
that does not have a dominating Makefile of it's own. And if all else 
fails you can just use the proposed plain project mark.

> What user-level commands are going to benefit from this setup?
It seems that we somewhat differ in our priorities for project 
treatment. You seem to prioritize the logical grouping of files for 
editing operations, while I prioritize "actionability". If a certain 
directory has a set of unique actions that can be performed on it, then 
it's a project for me. And while your observation that such emphasis on 
actionability may result in worse logical grouping is broadly true, it 
is not necessarily true that a blind reliance on VC backend would result 
in proper logical grouping. Sure, that would be true for most projects, 
but oftentimes you have multiple independent, but related projects 
sharing the same repository.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 06.10.2021 17:09, Nikolay Kudryavtsev wrote:
>> But when we go up a directory, "./project/". And when asked to list 
>> its files, how do we avoid including "./project/foo/a" in that list? 
>> It would make sense to exclude any nested projects, right?
> Not necessarily. It may be a useful thing to do for some projects, but 
> it does not follow from anything that it should be the only or the 
> default option.

When should we not do it?

> The correct solution here is IMHO implementing some kind 
> of .project-settings.el file in which you can set 
> hide-nested-project-files.

You can already specify ignored files in a project through .dir-locals.el.

But then you will end up specifying the same information twice. Once 
when setting up those new backends -- and the second time when 
configuring the parent project to ignore particular subdirectories. And 
that seems problematic from the usability standpoint. Also, that 
information can easily get out of sync.

>> Setting project-find-functions in a major mode is a questionable thing 
>> to do, because then you end up with Emacs where files in the same 
>> directory belong to different projects.
> I'm not talking about setting it locally in the mode, more about modes 
> providing such functions on load. That's another important question. 
> More backends are more functions to test, so it's reasonable to add 
> backends only when they're needed. I may avoid programming in language X 
> from some months so no reason to keep that backed on, but when I start 
> editing a file in that language, the major mode loads and so should the 
> backend.

If the only piece of information such modes intend to provide is the 
name, or multiple names, of "marker" files which indicate that their 
containing directory is the root, and if this information should be 
applied by the user manually anyway, why not have a single backend for 
that purpose, with a custom var containing the list of files names? The 
major modes can tell the users to modify that variable.

And this backend is in the proposed 'fallback' backend.

>> If there is a next backend which indicates the same root, why do we 
>> need the first one? 
> We don't know what the next backend in line indicates. Nor do we care 
> about it since the current one already gives us something. We just try 
> to find backends until we find one in the order of priority and then stop.

That didn't really answer my question.

>> Suppose I call project-find-file, meaning to jump to another file in 
>> the same Git repository. And instead I am shown only a list of files 
>> in the current subdirectory because it contains, say, a Makefile. Is 
>> that a good idea to enact this kind of behavior automatically? 
> If a user believes that it looks like a duck, it should squeak like one. 
> Sure you personally may want to suppress some possible project backends 
> from firing, but someone may want the opposite.

I'm not sure how I would suppress "possible project backends" from 
firing. That's the rub: the configuration is global, and whatever 
backend ends up being used, should be the most fitting to the total set 
of possible user's needs.

Meaning, it should correspond to the user's view of the project to the 
best possible ability: point out the root, exclude the files that need 
to be excluded, and ideally fetch the list of files quickly as well.

> I want to remind you of this recent-ish thread on HGE:
> 
> https://lists.gnu.org/archive/html/help-gnu-emacs/2021-09/msg00045.html
> 
> Lets take the maven example presented there. We have a project 
> containing modules. The user wants to compile both independently. We can 
> write two different compilation commands, one that works on the project 
> and one that works on the module. Or we can just have a single command, 
> since the compilation process is not different in any way for both. Then 
> we can give that command some prefix that would make it work not on the 
> project itself but on the root project of that project.

Either you have a compilation command that is Maven-aware (and thus can 
find the module root directory on its own), or we add an extension of 
the project API, with generic like project-modules, where modules behave 
like projects themselves (can implement the 'root' and 'files' methods). 
And maybe with a project-current-module generic as well, though it could 
easily be implemented with a linear search across a small list (with 
file-in-directory-p check).

But if the command creates a Maven-specific invocation, it doesn't need 
the above abstraction -- it works with Maven, so it knows Maven, it can 
look for the current module directly. Or just use the project root, if 
the appropriate value of prefix was specified.

I don't see where your approach would make things simpler/more 
predictable/reliable.

> The Makefile example is your strongest argument here, but we can define 
> find functions for it recursively. That is until you find a Makefile 
> that does not have a dominating Makefile of it's own. And if all else 
> fails you can just use the proposed plain project mark.

That's possible, but it's not at all a guarantee that in every big 
project every Makefile will have a "dominating" Makefile of its own.

The above is just a particular GNU convention. But Makefiles are also 
used for quick scripting in projects that are actually built with other 
tools.

>> What user-level commands are going to benefit from this setup?
> It seems that we somewhat differ in our priorities for project 
> treatment. You seem to prioritize the logical grouping of files for 
> editing operations, while I prioritize "actionability".

I'm prioritizing "universality", so to speak. And predictability. So 
that a certain class of commands (or several classes, actually) can use 
"current project" and be reliably certain of what it is.

> If a certain 
> directory has a set of unique actions that can be performed on it, then 
> it's a project for me.

It would seem like your vision of the project could benefit from a 
notion like "facet". E.g. a project lookup would search for not just 
"the current project", but "the current build project" or "current file 
listing project", or... I don't know what else, actually. But I'm sure 
there can be other additions (something test-framework related, maybe).

But unless I'm missing something major, the same goal could be served by 
an addition of a new hook. Like 'buildtool-find-functions'. Which would 
return, for example, new kind of object, and that object could tell the 
parent directory of its build file, and the list of the tasks described 
in it, and... perhaps something about how those tasks should be invoked. 
That new abstraction could be used by commands that want to interact 
with build files in an abstract fashion and to launch build tasks.

It might also have a problem choosing which Makefile to use, out of a 
hierarchy of Makefiles, though. Requiring similar customization capability.

> And while your observation that such emphasis on 
> actionability may result in worse logical grouping is broadly true, it 
> is not necessarily true that a blind reliance on VC backend would result 
> in proper logical grouping. Sure, that would be true for most projects, 
> but oftentimes you have multiple independent, but related projects 
> sharing the same repository.

There are two patches in this bug report. Have you looked at the other one?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Nikolay Kudryavtsev]

[-- Attachment #1: Type: text/plain, Size: 6336 bytes --]

> When should we not do it? 
Personal preference. I'd probably never hide subproject files in 99% of 
the projects I work on.

> But then you will end up specifying the same information twice. Once 
> when setting up those new backends -- and the second time when 
> configuring the parent project to ignore particular subdirectories.
You can have a `project-hide-nested-project-files' variable, set it once 
for all backends. Maybe per backend version too. For me personally, if I 
ever were to hide subproject files it would be on a rare project and it 
would be on the same backend, in which I would normally never hide them, 
so for my workflow this is a per project setting only.

> why not have a single backend for that purpose, with a custom var 
> containing the list of files names?
Right, so remember, I said we can use this to realize most of the 
Projectile backends? Well, the ones we can't realize would require 
regexps, usually of the "\\.ext$" kind. So you have to account for those 
too. Then there may be some other possible cases requiring custom function.

So we have at least two reasons to prefer adding actual separate 
backends over one catch-all filename based. Orderability and 
customizability.

Lets say I have this structure: VC root, subfolders containing multiple 
related, but independent projects in some programming language, backend 
for which exists; deeper in one of the subfolders I have a GTAGS file, 
assume GTAGS backend exists in one form or another. GTAGS file is placed 
in this location because elsewhere in the repo there are symbol names 
that are too close to each other and it's more convenient not having 
them show up when I lookup something. I don't want VC backend to define 
the project root here for obvious reasons. The major mode build tool 
backend should do OK and I may or may not want GTAGS to define the 
project root.

Having one find-function list which the user can reorder as he sees fit 
and that list may contain not just filenames, but regexps and custom 
functions too.

As for customizability: we're already discussing at least 2 backend 
settings: hide-nested-project-files and recursivity. Those settings 
require a backend to be something more than a filename string in a 
secondary list.

> That didn't really answer my question. 
All right, lets rephrase the answer. At the moment in time a backend is 
defined we do not know every single exact situation that backend would 
have to operate in, because that would require the ability to predict 
time, which we technologically do not have at the current moment. Lets 
say I add a backend for my major mode. Someone in exactly 18 days, 6 
hours, 5 minutes and 3 seconds decides to use it to work on his project. 
Unfortunately I do not know whether his project is in VCd and if VC 
project backend returns the same root. If I could predict time and my 
prediction of all possible future use cases would show that VC backend 
returns the same root for every single one of them, I of course would 
not bother adding mine. But because my imperfect human understanding 
makes me think that it won't, I have to add a backend of my own.

> I'm not sure how I would suppress "possible project backends" from 
> firing. 
Since you believe that VC backend gives you the best result, you can 
always ensure on your end that you never get, say a Makefile backend in 
your project-find-functions.

Probably the best route here globally, is changing 
project-find-functions to a list with numerical priorities, so that you 
could set VC backend to priority 0 in your init and instruct mode 
developers to never put anything with the same priority or higher in the 
docstring.

> That's possible, but it's not at all a guarantee that in every big 
> project every Makefile will have a "dominating" Makefile of its own. 
Yes, but we can define a list of possible parent backends for every 
backend. For example you could set VC as possible parent backend for 
Makefile. Would probably not be a good idea in general, but for your 
VC-first workflow, should be fine.

> It would seem like your vision of the project could benefit from a 
> notion like "facet". E.g. a project lookup would search for not just 
> "the current project", but "the current build project" or "current 
> file listing project", or... I don't know what else, actually. But I'm 
> sure there can be other additions (something test-framework related, 
> maybe). 
Or a "module" right? I was thinking about this too, and could not find a 
good name for it either. Such a solution would be a reliable working 
compromise between our schools of thought. You get your project-root 
untouched, I get my own project root to do whatever I please with it. 
The problem with it is that it's really overengineered. For most 
projects there would be a 1 to 1 relationship between the project and 
the module(artifact?) and even if it's not 1 to 1, there's a root module 
most of the time. That's why it feels inferior to me in comparison to 
just treating everything as projects and going bottom up.

> Which would return, for example, new kind of object, and that object 
> could tell the parent directory of its build file, and the list of the 
> tasks described in it, and... perhaps something about how those tasks 
> should be invoked. That new abstraction could be used by commands that 
> want to interact with build files in an abstract fashion and to launch 
> build tasks. 
After studying Projectile build commands I found them inadequate, since 
it provides just two, compile and build, while even my simple major mode 
requires a separate debugging command too. This solution suffers from 
the same lack of flexibility. Take for example unit testing. It's not 
necessarily build tool subservient and can be independent from it, but 
it is situated in relation to the build tool root. The maven hierarchy 
is problematic for this too, while my "treat everything as projects" 
paradigm has an elegant solution in which you can use a numerical prefix 
to launch a build command on the Nth parent of the current project.

> There are two patches in this bug report. Have you looked at the other 
> one?
You mean the original patch? Well it is IMHO better than yours since 
it's less ambitious and does not go in what I believe to be the wrong 
direction.


[-- Attachment #2: Type: text/html, Size: 7783 bytes --]

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 07.10.2021 16:08, Nikolay Kudryavtsev wrote:
>> When should we not do it? 
> Personal preference. I'd probably never hide subproject files in 99% of 
> the projects I work on.

All right. So you might like the possible alternative approach to this 
problem.

What's your stance of having "filemarker" projects honor (or not honor) 
.gitignore instructions from the containing VCS repo?

>> But then you will end up specifying the same information twice. Once 
>> when setting up those new backends -- and the second time when 
>> configuring the parent project to ignore particular subdirectories.
> You can have a `project-hide-nested-project-files' variable, set it once 
> for all backends. Maybe per backend version too.

The problem is not being able to invent a variable, but how to honor it 
when listing project files. If the same backend doesn't have the 
information about the contained projects.

>> why not have a single backend for that purpose, with a custom var 
>> containing the list of files names?
> Right, so remember, I said we can use this to realize most of the 
> Projectile backends? Well, the ones we can't realize would require 
> regexps, usually of the "\\.ext$" kind. So you have to account for those 
> too. Then there may be some other possible cases requiring custom function.

project-fallback-markers can easily support globs (I didn't add the 
support initially because the naive implementation of checking for files 
including globs is slower). Support for predicate functions among its 
entries is pretty trivial to do.

OTOH, when we only have a list of project-finding functions (one per 
project marker, say), it's pretty difficult to find the deepest 
enclosing project directory without running them all, and having them 
traverse the parent directories up until /. This is an inevitably slow 
approach. Can be bearable in a local filesystem; might be fatal over Tramp.

> Lets say I have this structure: VC root, subfolders containing multiple 
> related, but independent projects in some programming language, backend 
> for which exists; deeper in one of the subfolders I have a GTAGS file, 
> assume GTAGS backend exists in one form or another. GTAGS file is placed 
> in this location because elsewhere in the repo there are symbol names 
> that are too close to each other and it's more convenient not having 
> them show up when I lookup something. I don't want VC backend to define 
> the project root here for obvious reasons. The major mode build tool 
> backend should do OK and I may or may not want GTAGS to define the 
> project root.

It doesn't seem like you want GTAGS to define project root; you'll only 
want certain commands, like xref-find-definitions, to use the closes 
GTAGS file.

Good thing xref-find-definitions doesn't use the project.el 
infrastructure at all.

> Having one find-function list which the user can reorder as he sees fit 
> and that list may contain not just filenames, but regexps and custom 
> functions too.

I'm not seeing the concrete usage scenarios yet.

> As for customizability: we're already discussing at least 2 backend 
> settings: hide-nested-project-files and recursivity. Those settings 
> require a backend to be something more than a filename string in a 
> secondary list.

These settings, as I see it, will be on the project-vc backend.

>> That didn't really answer my question. 
> All right, lets rephrase the answer. At the moment in time a backend is 
> defined we do not know every single exact situation that backend would 
> have to operate in, because that would require the ability to predict 
> time, which we technologically do not have at the current moment. Lets 
> say I add a backend for my major mode. Someone in exactly 18 days, 6 
> hours, 5 minutes and 3 seconds decides to use it to work on his project. 
> Unfortunately I do not know whether his project is in VCd and if VC 
> project backend returns the same root. If I could predict time and my 
> prediction of all possible future use cases would show that VC backend 
> returns the same root for every single one of them, I of course would 
> not bother adding mine. But because my imperfect human understanding 
> makes me think that it won't, I have to add a backend of my own.

Okay, but we need both correctness and speed. Answering "see if there is 
a fast backend behind this one with the same root" is not very useful, 
since in this scenario we seemingly (!) don't need the first backend anyway.

But perhaps you meant we should always scan the full list of backends, 
and we should not only accept a "fast" backend with the same root, but 
with root in any parent directory as well. Which would correspond to the 
popular situation with the monorepo.

Setting aside the performance concerns of always scanning for all 
backends in project-find-functions, what do we do with the ignore 
entries? A backend is (very roughly) defined as (root . ignores), to be 
able to only list and work with a subset of files from its root's 
directory tree.

The VC backend traditionally includes .gitignore entries in its list of 
ignores. So its project-ignores method includes them, and its 
project-files method implicitly uses them.

But what if your first detected backend has a different list of ignores 
configured? Or none? Do we account for the latter's (fast) backend 
ignores when asking it for the list of files in the first backend's 
detected root directory. In practical terms, as just one example, that 
would mean honoring .gitignore at the top of a monorepo. Which might 
seem like a good thing to do, but also a somewhat unexpected behavior, 
conceptually.

> Probably the best route here globally, is changing 
> project-find-functions to a list with numerical priorities, so that you 
> could set VC backend to priority 0 in your init and instruct mode 
> developers to never put anything with the same priority or higher in the 
> docstring.

That's quite easy to do already: project-find-functions is a hook, and 
'add-hook' has a DEPTH argument. Since Emacs 27, I think.

>> That's possible, but it's not at all a guarantee that in every big 
>> project every Makefile will have a "dominating" Makefile of its own. 
> Yes, but we can define a list of possible parent backends for every 
> backend. For example you could set VC as possible parent backend for 
> Makefile. Would probably not be a good idea in general, but for your 
> VC-first workflow, should be fine.

I think it's apparent at this point that we are venturing into the 
territory of pretty invasive, backward-incompatible changes to the 
existing project.el API.

>> It would seem like your vision of the project could benefit from a 
>> notion like "facet". E.g. a project lookup would search for not just 
>> "the current project", but "the current build project" or "current 
>> file listing project", or... I don't know what else, actually. But I'm 
>> sure there can be other additions (something test-framework related, 
>> maybe). 
> Or a "module" right? I was thinking about this too, and could not find a 
> good name for it either.

Right, yeah. Having module detection on a hook seems more flexible than 
putting it on a method dispatches by project type, because then the 
Maven extension can work with any project backend, not just some 
specific Maven-supporting one.

> Such a solution would be a reliable working 
> compromise between our schools of thought. You get your project-root 
> untouched, I get my own project root to do whatever I please with it. 
> The problem with it is that it's really overengineered. For most 
> projects there would be a 1 to 1 relationship between the project and 
> the module(artifact?) and even if it's not 1 to 1, there's a root module 
> most of the time. That's why it feels inferior to me in comparison to 
> just treating everything as projects and going bottom up.

I don't know if it's really overengineered, especially compared to 
certain proposals in your previous email.

FWIW, Steinar asked for being able to choose whether to act on a module 
or on a project, and for that such separation seems to be necessary.

Thanks for the reminder about that thread, BTW.

Speaking of build tools -- just as well, if the tool is in the same 
directory as the project root, you don't need any additional backends. 
But if you do need extra detection, file-finding logic, that could be 
put into buildtool-find-functions.

>> Which would return, for example, new kind of object, and that object 
>> could tell the parent directory of its build file, and the list of the 
>> tasks described in it, and... perhaps something about how those tasks 
>> should be invoked. That new abstraction could be used by commands that 
>> want to interact with build files in an abstract fashion and to launch 
>> build tasks. 
> After studying Projectile build commands I found them inadequate, since 
> it provides just two, compile and build, while even my simple major mode 
> requires a separate debugging command too.

Debugging seems to be a separate feature from build tools (requiring its 
own information, and a fair amount of it). Although, depending on a 
language and environment, it might require integration with build tools 
as well.

> This solution suffers from 
> the same lack of flexibility. Take for example unit testing. It's not 
> necessarily build tool subservient and can be independent from it, but 
> it is situated in relation to the build tool root.

So what's the problem with this approach, then? The run-tests command 
can for example run the buildtool-find-functions, take the appropriate 
one, search its parent directory for the presence of testing frameworks, 
and launch it.

> The maven hierarchy 
> is problematic for this too, while my "treat everything as projects" 
> paradigm has an elegant solution in which you can use a numerical prefix 
> to launch a build command on the Nth parent of the current project.

It sounds like this approach, at the very least, doesn't go through 
'project-current'?

Are there other advantages, rather than being able to choose the project 
parent nesting depth with the prefix argument?

>> There are two patches in this bug report. Have you looked at the other 
>> one?
> You mean the original patch? Well it is IMHO better than yours since 
> it's less ambitious and does not go in what I believe to be the wrong 
> direction.

The 'project-vc-subprojects' patch. Here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#85

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Nikolay Kudryavtsev]

> What's your stance of having "filemarker" projects honor (or not 
> honor) .gitignore instructions from the containing VCS repo? 
That's a good question. I think at any non-VC project it is fine that it 
does not honor it, since you get what you've signed up for. I'll come 
back to this later in this letter, since another part of this 
conversation sort of gives the answer to it.

> The problem is not being able to invent a variable, but how to honor 
> it when listing project files. If the same backend doesn't have the 
> information about the contained projects. 
Yes, but at this point it's a separate feature request for a problem 
that already exists at this point, it's just that we rely on other tools 
to solve it for us. Let's say I maintain an application, except for a 
module maintained by my colleague Bob. And since that module may be be 
shared by multiple applications, it is version controlled separately. I 
also may or may not want it to be considered a part of my project for 
the sake of project utility commands. Here we rely on .gitignore and the 
like. Lets say with GTAGS I can decide whether to include the bob module 
too. But all of this works only because those backended tools happen to 
implement this ignore functionality. Lets say your VC is broken and just 
won't ignore some files(happend before in practice with VC) even though 
it ordered to? So any such project-level ignore functionality, while 
useful and would be even more useful with more weaker custom backends, 
is not that important in the context of this discussion in my opinion.

> This is an inevitably slow approach. Can be bearable in a local 
> filesystem; might be fatal over Tramp. 
Yes. That's sort of the unfortunate price we have to pay here. I think 
some caching can be implemented, to limit directory reading operations 
to 1 per level.

> It doesn't seem like you want GTAGS to define project root; you'll 
> only want certain commands, like xref-find-definitions, to use the 
> closes GTAGS file. 
If I've made a GTAGS file at this level with the explicit intention to 
exclude everything else I might as well be be interested in running 
project-find-regexp and the like here, right?

> I'm not seeing the concrete usage scenarios yet. 
Well think more about either the 3 marker(vc, build-tool, GTAGS) example 
or the Bob module example. Lets try to put it in your terms of 
"correctness". Is me wanting the build tool to define the project 
boundary over VC "correct"? Is me possibly wanting GTAGS backend to 
define the project boundary "correct"? Is me wanting to possibly NOT 
exclude the Bob module from the project "correct"? Lets say I've found a 
broken function in the general application and I now want to see whether 
the Bob module uses it too along with any other code by calling 
project-find-regexp. You may answer no to all of those and say that Holy 
Correctness can be only provided by the Holy VC backend, but I don't 
think it's a reasonable approach.

> These settings, as I see it, will be on the project-vc backend.
Are you saying that if I want to declare that my make projects can be 
subordinate to another build tool, I have to declare that in the 
project-vc-backend of all places?

> Okay, but we need both correctness and speed.
Lets for a moment assume that my use cases are "correct". Meaning that 
relying on VC here would be doing the wrong thing fast. Is doing the 
wrong thing fast a good idea?

> Answering "see if there is a fast backend behind this one with the 
> same root" is not very useful, since in this scenario we seemingly (!) 
> don't need the first backend anyway. 
There are two possible scenarios.

A. Backend project marker has the same root as VC.

B. Backend project marker has a different root.

You look ONLY at the scenario A and say "why do we need a new backend"? 
Well we need it ONLY for the scenario B, which you are ignoring.

Again, not gonna repeat, myself on the ignore issue, apart from "you get 
what you've signed up for".

> I think it's apparent at this point that we are venturing into the 
> territory of pretty invasive, backward-incompatible changes to the 
> existing project.el API. 
True, but currently the project.el is underutilized due to its 
simplicity and due to this we can still do reasonable invasive changes 
to the API.

> Right, yeah. Having module detection on a hook seems more flexible 
> than putting it on a method dispatches by project type, because then 
> the Maven extension can work with any project backend, not just some 
> specific Maven-supporting one. 
The possible actions for Maven would already work any project backend, 
it's the question of how to ensure the proper root for it.
> FWIW, Steinar asked for being able to choose whether to act on a 
> module or on a project, and for that such separation seems to be 
> necessary. 

I don't think he cares about Emacs having a distinction, more about the 
abiilty to independently act on(compile) those parts.

> Speaking of build tools -- just as well, if the tool is in the same 
> directory as the project root, you don't need any additional backends. 
> But if you do need extra detection, file-finding logic, that could be 
> put into buildtool-find-functions.
That's the thing, you would always need one.

> Debugging seems to be a separate feature from build tools (requiring 
> its own information, and a fair amount of it). Although, depending on 
> a language and environment, it might require integration with build 
> tools as well.
Yeah, and that's one of the conceptual problems with that approach: 
assuming too much. Project markers(or "build-tools" markers in your 
paradigm) are not limited to build tools, it may be a tags tool for example.
> So what's the problem with this approach, then?
The main problem is that it's conceptually wrong IMHO. In trying to make 
anything not VC have a subservient status you're actually elevating it 
to a special one. Not any project backend needs build-tools, for example 
possible GTAGS won't, while it is quite fine that any project is aware 
of its VC regardless of the backend. So the user defined backends should 
be given primacy and VC should enjoy that (elevated secondary 
subservient) special status, with the fallback VC backend staying there 
too of course. And that's also the answer to your original question 
about honoring .gitignore. If VC is always available regardless of the 
backend, we can surely honor it and also use it for whichever 
optimizations are possible.

Even take Zhou's patch. Lets operate on the assumption that we cannot 
let anyone go astray from the paradise of the Holy VCs correctness. 
Every single problem you've mentioned would befall on those trying to 
use a plain project. Adding a new backend would just let the heathens 
win, since they can add those files anywhere and stray away from the 
light. From such point of view, you can never add new backends, you can 
only add new build tools.

Or to put it another way, whenever people want to mess with project 
backends, what they want is to define a scope in which they want to 
operate. That scope may differ from the scope defined by VC. Also 
there's nothing wrong in VC scope being available at the same time at 
the user declared project scope. What I mean by that is that nobody 
would complain if every project-* utility commands gets a sister named 
project-vc-* that would allow you to the same operation, but explicitly 
within the VC scope of the current project.

> The 'project-vc-subprojects' patch.
This a solution to the subprojects hiding problem discussed earlier, 
perfectly acceptable for me, but as I mentioned before is not something 
I'd use much personally.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 08.10.2021 19:24, Nikolay Kudryavtsev wrote:
>> What's your stance of having "filemarker" projects honor (or not 
>> honor) .gitignore instructions from the containing VCS repo? 
> That's a good question. I think at any non-VC project it is fine that it 
> does not honor it, since you get what you've signed up for. I'll come 
> back to this later in this letter, since another part of this 
> conversation sort of gives the answer to it.

I didn't find a definite answer in the rest of the email, so I'll 
continue this thread here.

This is the problem: semantically, it makes sense for filemarker 
backends not to honor VC ignores. But in practice, people will often 
(probably most often) want it to, or implicitly expect it to.

Because when you are working on a subproject in a big monorepo, you 
might want to focus on its directory tree, sure, but you also expect 
that the global .gitignore settings are honored (where build artefacts, 
temporary files, etc, are configured to be skipped). You might also have 
additional per-directory .gitignore files which, again, are even more 
pertinent to the subproject, but semantically would not be expected to 
be honored.

It's not good to have conflicts like that, between semantics and 
practical requirements.

>> The problem is not being able to invent a variable, but how to honor 
>> it when listing project files. If the same backend doesn't have the 
>> information about the contained projects. 
> Yes, but at this point it's a separate feature request for a problem 
> that already exists at this point, it's just that we rely on other tools 
> to solve it for us. Let's say I maintain an application, except for a 
> module maintained by my colleague Bob. And since that module may be be 
> shared by multiple applications, it is version controlled separately. I 
> also may or may not want it to be considered a part of my project for 
> the sake of project utility commands. Here we rely on .gitignore and the 
> like. Lets say with GTAGS I can decide whether to include the bob module 
> too. But all of this works only because those backended tools happen to 
> implement this ignore functionality.

So you want to add a new backend based on GTAGS which also lists files 
by calling 'global -P' or something. It's not enough to configure the 
root-finding logic for it, you also need to implement 'project-files' 
with the above process call.

I also wouldn't want to enable such backend by default because it will 
only list files from a manually generated index, but not, say, a file 
that the user just created and saved. Not very user-friendly.

But enable or not, that's beside the point: such backend fits the 
current approach well; whoever writes it can distribute it, and the 
users who want to prioritize GTAGS over .git can put it in front.

It doesn't seem pertinent to what project-fallback should do.

> Lets say your VC is broken and just 
> won't ignore some files(happend before in practice with VC) even though 
> it ordered to? So any such project-level ignore functionality, while 
> useful and would be even more useful with more weaker custom backends, 
> is not that important in the context of this discussion in my opinion.

It doesn't really match my experience. Some VCS can be "broken" -- then 
we can ignore their equivalent of 'git ls-files' and use 'find'. We only 
have optimized paths for Git and Hg anyway. But even when we use 'find', 
we try to pick up the ignores configuration from the repo.

>> This is an inevitably slow approach. Can be bearable in a local 
>> filesystem; might be fatal over Tramp. 
> Yes. That's sort of the unfortunate price we have to pay here. I think 
> some caching can be implemented, to limit directory reading operations 
> to 1 per level.

"Some caching" has been lying around in TODO for years.

>> I'm not seeing the concrete usage scenarios yet. 
> Well think more about either the 3 marker(vc, build-tool, GTAGS) example 
> or the Bob module example. Lets try to put it in your terms of 
> "correctness". Is me wanting the build tool to define the project 
> boundary over VC "correct"? Is me possibly wanting GTAGS backend to 
> define the project boundary "correct"? Is me wanting to possibly NOT 
> exclude the Bob module from the project "correct"? Lets say I've found a 
> broken function in the general application and I now want to see whether 
> the Bob module uses it too along with any other code by calling 
> project-find-regexp. You may answer no to all of those and say that Holy 
> Correctness can be only provided by the Holy VC backend, but I don't 
> think it's a reasonable approach.

I want all of those capabilities to be reachable, but we also need to 
decide which project backends configuration is on by default.

>> These settings, as I see it, will be on the project-vc backend.
> Are you saying that if I want to declare that my make projects can be 
> subordinate to another build tool, I have to declare that in the 
> project-vc-backend of all places?

What is a 'make project'? If you have 'Makefile' inside a subdirectory 
of a GGTAGS project, do you expect the said 'make project' only to 
include the files that GGTAGS has indexed?

If yes, you need to have to integration with GGTAGS, one way or another. 
Maybe simply go through the GGTAGS backend to configure said 'make 
project', since that is the backend which is expected to honor the 
GGTAGS index.

If no, sure, have a 'make project' entry in front of 'ggtags project' 
entry in project-find-functions. That fits the current approach well.

> Again, not gonna repeat, myself on the ignore issue, apart from "you get 
> what you've signed up for".

And going back to the ignores issue: I think we shouldn't expect the 
users to know the intricacies and the semantics of the project backend 
configuration before they can use the corresponding features 
productively. The default config should fit the majority use cases in a 
predictable way.

OTOH, when you are well-acquainted with the nuts and bolts, you can 
built your own backend configuration, and you wouldn't care about the 
default setup. For such a user the current project-fallback definition 
is unnecessary: it is pretty trivial after all.

>> I think it's apparent at this point that we are venturing into the 
>> territory of pretty invasive, backward-incompatible changes to the 
>> existing project.el API. 
> True, but currently the project.el is underutilized due to its 
> simplicity and due to this we can still do reasonable invasive changes 
> to the API.

Not sure if that's true. According to the feedback, it covers the use 
cases of like 80% of the users. And the patches in this discussions 
should cover the remaining big holes which have been reported a few times.

So... I'm fine discussing additions and even big remodels, but it would 
be better to consider those piece-by-piece, and probably in separate bug 
reports. We should still keep an eye out for performance, though, and 
the OOtB experience.

>> Right, yeah. Having module detection on a hook seems more flexible 
>> than putting it on a method dispatches by project type, because then 
>> the Maven extension can work with any project backend, not just some 
>> specific Maven-supporting one. 
> The possible actions for Maven would already work any project backend, 
> it's the question of how to ensure the proper root for it.

Yes.

>> FWIW, Steinar asked for being able to choose whether to act on a 
>> module or on a project, and for that such separation seems to be 
>> necessary. 
> 
> I don't think he cares about Emacs having a distinction, more about the 
> abiilty to independently act on(compile) those parts.

If the user wants to choose what to act on (whole project or current 
module), the information about modules needs to be discovered as a 
separate semantic unit. Not sure how else it would be possible to 
implement such choice in user interaction.

>> Debugging seems to be a separate feature from build tools (requiring 
>> its own information, and a fair amount of it). Although, depending on 
>> a language and environment, it might require integration with build 
>> tools as well.
> Yeah, and that's one of the conceptual problems with that approach: 
> assuming too much. Project markers(or "build-tools" markers in your 
> paradigm) are not limited to build tools, it may be a tags tool for 
> example.

Only if you intend to use the 'tags tool' as a build tool, e.g. list the 
available tasks or invoke one (for example, to rebuild the index). Then 
said tags tool can be put into build-tools-functions, with appropriate 
implementations for 'list tasks' and 'run task'.

> Even take Zhou's patch. Lets operate on the assumption that we cannot 
> let anyone go astray from the paradise of the Holy VCs correctness. 
> Every single problem you've mentioned would befall on those trying to 
> use a plain project. Adding a new backend would just let the heathens 
> win, since they can add those files anywhere and stray away from the 
> light. From such point of view, you can never add new backends, you can 
> only add new build tools.

You can add new backends: you can add a bridge to Projectile, for 
example (there is one inside one of the reports in its issue tracker). 
Or you can create the backend based on GGTAGS, the one you described.

As long as it's feature-rich enough, and the behaviors are documented, 
so the users know what to expect from it, it's totally fine.

But both of those are more complex than the project-plain backend Zhou 
originally proposed, and the project-fallback in the latest patch.

> Or to put it another way, whenever people want to mess with project 
> backends, what they want is to define a scope in which they want to 
> operate. That scope may differ from the scope defined by VC.

I'd like to point out that this is already easy to do.

> Also 
> there's nothing wrong in VC scope being available at the same time at 
> the user declared project scope. What I mean by that is that nobody 
> would complain if every project-* utility commands gets a sister named 
> project-vc-* that would allow you to the same operation, but explicitly 
> within the VC scope of the current project.

Do we really want to ask everyone to use a separate set of commands to 
apply the 'ignores' config from the current VCS repo?

I mean, it's nice to have the option for certain cases, but one would 
imagine this behavior would be more desirable by default, no?

>> The 'project-vc-subprojects' patch.
> This a solution to the subprojects hiding problem discussed earlier, 
> perfectly acceptable for me, but as I mentioned before is not something 
> I'd use much personally.

FWIW, a user option 'do not exclude subprojects' could very easily be 
added as well.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Nikolay Kudryavtsev]

> I didn't find a definite answer in the rest of the email, so I'll 
> continue this thread here. 
Lets restate the answer: VC should be available to the user regardless 
of the current project backend. When it is so, you can use whatever 
defaults you deem reasonable, with ignores being respected probably 
being a good choice.

> So you want to add a new backend based on GTAGS which also lists files 
> by calling 'global -P' or something.
> I want all of those capabilities to be reachable, but we also need to 
> decide which project backends configuration is on by default.
The question of default project backends is pretty irrelevant here. I'm 
not suggesting adding anything there. I'm just saying that when the user 
adds something, it should not be reduced to the second class citizenship.

The reason I've jumped in against your patch in particular is because 
that's what it does. It makes a distinction between the (possibly) full 
fledged function backends and the inferior marker file backends, which 
get their own little ghetto. While I believe that we should treat such 
marker file backends as first class citizens that are equal to any other 
backend.

> If the user wants to choose what to act on (whole project or current 
> module), the information about modules needs to be discovered as a 
> separate semantic unit.
  I don't think so. We have two options:

1. Nestable projects. In this paradigm any project can have however many 
other projects in each subdirectories. You can go from child project to 
a parent project pretty easily. Going from parent project to child 
projects is a little bit harder, but still possible. So I can 
compile(use custom action on) the parent projects from a child one, or a 
particular(or all) child projects from parent.

2. Project units. Not gonna use the word "module" here, because we've 
already used it for a different thing in this discussion. So one project 
can have 0 or more units. Units cannot exist outside of a project and 
are in turn are not projects themselves. Except that they can completely 
look like projects, in cases like Maven or Makefiles.

Option 1 is simpler because it has only 1 concept. Option 2 introduces a 
second separate semantic unit, but I don't see any benefit from 
introducing it.

> Only if you intend to use the 'tags tool' as a build tool, e.g. list 
> the available tasks or invoke one (for example, to rebuild the index). 
> Then said tags tool can be put into build-tools-functions, with 
> appropriate implementations for 'list tasks' and 'run task'.
I don't like such over-engineering, due to conceptually assuming that 
all projects are build tool projects, all actions being connected to 1 
particular tool and each tool necessarily having multiple actions. I 
believe all of this is better left to users and backend developers. With 
just actions and maybe action groups, if we want them.

> Do we really want to ask everyone to use a separate set of commands to 
> apply the 'ignores' config from the current VCS repo? 
No? Separate commands is just another advantage of VC being always 
there(if it's there). Maybe I'm maintaining a certain part of the 
repository, be it a module or whatever, and that's why I'm treating that 
part as a project, since that's the scope I normally care about. But 
then I notice a buggy function and use those commands to look whether 
anything else apart from my code uses it in the wider repo.

P.S. Sorry for the terrible amount of typing errors in my previous 
letter, too much writing for me last week.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 11.10.2021 21:05, Nikolay Kudryavtsev wrote:
>> I didn't find a definite answer in the rest of the email, so I'll 
>> continue this thread here. 
> Lets restate the answer: VC should be available to the user regardless 
> of the current project backend. When it is so, you can use whatever 
> defaults you deem reasonable, with ignores being respected probably 
> being a good choice.

Available how? Through a certain keymap, or a specially named set of 
commands? These issues matter.

>> So you want to add a new backend based on GTAGS which also lists files 
>> by calling 'global -P' or something.
>> I want all of those capabilities to be reachable, but we also need to 
>> decide which project backends configuration is on by default.
> The question of default project backends is pretty irrelevant here.

If you don't care about the defaults, why argue on this bug tracker at 
all? By similar logic, you can just go and write your own backends/apis/etc.

> I'm 
> not suggesting adding anything there. I'm just saying that when the user 
> adds something, it should not be reduced to the second class citizenship.

But it's not reduced: quite the opposite, whatever the user adds (to the 
beginning of the list) takes up a significant responsibility. That can 
be tough, however, so the first recommendation is to avoid doing it.

And solve the immediate goals using something else.

> The reason I've jumped in against your patch in particular is because 
> that's what it does. It makes a distinction between the (possibly) full 
> fledged function backends and the inferior marker file backends, which 
> get their own little ghetto. While I believe that we should treat such 
> marker file backends as first class citizens that are equal to any other 
> backend.

They are treated just as any backend in the list. According to its 
modest capabilities, however, the backend is placed at the end.

It's not the only possible solution to the problem, but I have yet to 
see a complete design of some alternative being presented.

>> If the user wants to choose what to act on (whole project or current 
>> module), the information about modules needs to be discovered as a 
>> separate semantic unit.
>   I don't think so. We have two options:
> 
> 1. Nestable projects. In this paradigm any project can have however many 
> other projects in each subdirectories. You can go from child project to 
> a parent project pretty easily. Going from parent project to child 
> projects is a little bit harder, but still possible.

Possible how?

> So I can 
> compile(use custom action on) the parent projects from a child one, or a 
> particular(or all) child projects from parent.
> 
> 2. Project units. Not gonna use the word "module" here, because we've 
> already used it for a different thing in this discussion. So one project 
> can have 0 or more units. Units cannot exist outside of a project and 
> are in turn are not projects themselves. Except that they can completely 
> look like projects, in cases like Maven or Makefiles.
> 
> Option 1 is simpler because it has only 1 concept. Option 2 introduces a 
> second separate semantic unit, but I don't see any benefit from 
> introducing it.

For instance, Maven modules can be reliably discovered from the parent 
project's root directory, unlike the potential discovery logic for 
arbitrarily nested projects (which seems like it will require a costly 
directory tree walk).

>> Only if you intend to use the 'tags tool' as a build tool, e.g. list 
>> the available tasks or invoke one (for example, to rebuild the index). 
>> Then said tags tool can be put into build-tools-functions, with 
>> appropriate implementations for 'list tasks' and 'run task'.
> I don't like such over-engineering, due to conceptually assuming that 
> all projects are build tool projects, all actions being connected to 1 
> particular tool and each tool necessarily having multiple actions. I 
> believe all of this is better left to users and backend developers. With 
> just actions and maybe action groups, if we want them.

Over-engineering?

It was just an example for what a "build tool API" could look like. If 
you know better, try to propose a different one.

>> Do we really want to ask everyone to use a separate set of commands to 
>> apply the 'ignores' config from the current VCS repo? 
> No? Separate commands is just another advantage of VC being always 
> there(if it's there). Maybe I'm maintaining a certain part of the 
> repository, be it a module or whatever, and that's why I'm treating that 
> part as a project, since that's the scope I normally care about. But 
> then I notice a buggy function and use those commands to look whether 
> anything else apart from my code uses it in the wider repo.

I meant a separate set of commands which would apply VCS's ignores to 
the project operations, which the "current" VC-unrelated backend 
apparently might not do, according to how I understood your explanations.

Overall, the idea about a separate keymap that will allow the user to 
act on the "parent" project is valid. Though it can be implemented on 
top of the current approach just as well.

Treating Maven modules as separate sub-projects *might* fit well with 
that as well.

But the overall idea of having lots of backends in the list, each 
describing a particular project type (marked by a particular filename) 
still look problematic to me from the practical standpoint (1. lower 
performance, 2. the problem of following .gitignore and the associated 
semantics seems unsolved).

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Nikolay Kudryavtsev]

> Available how? Through a certain keymap, or a specially named set of 
> commands? These issues matter. 
Available through a function project-current-vc. On top of that you can 
implement (global, per backend, per project) setting 
project-honour-vc-ignores. Then project-vc-* utility functions.

> If you don't care about the defaults, why argue on this bug tracker at 
> all?
Because when you're discussing multiple complicated interconnected 
issues, it's nice to spot one you don't have to discuss.

> But it's not reduced: quite the opposite, whatever the user adds (to 
> the beginning of the list) takes up a significant responsibility.
It is relegated to the second class citizenship. Remember, here I was 
talking about your patch in particular. It puts file marker backends 
into a separate list, not the generic case of users adding new backends.

> It's not the only possible solution to the problem, but I have yet to 
> see a complete design of some alternative being presented.
I've already presented what I'm proposing in previous emails:

(register-project-backend "project.file" priority 
:optimizes-file-listing nil :honuor-vc-ignores t)

Instead of adding any separate lists the same marker find functionality 
can be implemented in a backend registration function and this would 
allow people needing file marker backends to implement them on the fly, 
whenever they need one. Also it's not touching the defaults in any way, 
leaving the decision about the backend priority to the user.

> For instance, Maven modules can be reliably discovered from the parent 
> project's root directory, unlike the potential discovery logic for 
> arbitrarily nested projects (which seems like it will require a costly 
> directory tree walk). 
Yes. At the basic level it's a directory tree walk. But that's not a 
drawback of my particular proposal, since any attempt to implement the 
project unit functionality would suffer from this, regardless of the 
design chosen. You can implement any discovery strategy with any design.

> It was just an example for what a "build tool API" could look like. If 
> you know better, try to propose a different one. 
I've already proposed one. Each project backend has a set of actions. An 
action is just a function. Since most would probably be simple shell 
commands in the project-root, some easier way to define those should 
implemented. You can see whichever actions are available in some 
interactive dispatch function or the menu. Actions can be grouped into 
action groups. This is better in my opinion due to not presuming 
anything, so if you need to group by a build tool or by multiple, you 
can, but you don't have to.

> I meant a separate set of commands which would apply VCS's ignores to 
> the project operations, which the "current" VC-unrelated backend 
> apparently might not do, according to how I understood your explanations.
That would a bit clunky. Just honoring VC ignores by default is much 
better, with an option to do the opposite.

> Treating Maven modules as separate sub-projects *might* fit well with 
> that as well. 
If you can act on parent and child projects of the current, there's even 
less of a point to implement project units as a separate semantic 
unit(concept) from the project itself.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

Another issue is people working on monorepos (often backed by Git) 
sometimes want to split them into separate projects.

Either because they only work on a certain part of the whole project, or 
because they want to use Eglot, and that package has tied LSP roots 
detection to project roots (but the language server might expect some 
project configuration files in the root which reside in a subdirectory).

Here's a typical question/request for this functionality: 
https://emacs.stackexchange.com/questions/58463/project-el-override-project-root-with-dir-local-var/58468

Patch attached, it adds new user option project-vc-subprojects which 
alters the root-finding logic and cuts out the subproject contents from 
the parent project with the mechanism of "ignores".

The latter capability (excluding subproject's files) informed the choice 
of the approach. Which is altering the vc project backend's behavior, 
rather that offering this feature through another backend, like the one 
added in the previous patch, for example.

I don't know if all users of this feature will want them excluded, 
though. The attached implementation does, and maybe another option could 
be added to disable this.

Or we could drop this part of the behavior, insisting that users who 
want it could add the corresponding entries to project-vc-ignores. This 
way they would be listing the subprojects twice, however. And the 
project-vc-merge-submodules=nil behavior matches the other option 
(submodule files are excluded from the parent).

Again, thoughts welcome.

[-- Attachment #2: project-vc-subprojects.diff --]
[-- Type: text/x-patch, Size: 3420 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 57a961c260..e45667ed15 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -369,6 +369,23 @@ project-vc-merge-submodules
   :package-version '(project . "0.2.0")
   :safe #'booleanp)
 
+(defcustom project-vc-subprojects nil
+  "List of relative directory names to consider separate projects.
+Each entry should a string, name of a subproject root directory
+relative to the VC root.
+
+Every entry in this list will be considered a separate project
+for the purposes of files listings, searches, etc, and the parent
+project will exclude those files.
+
+One would usually set this variable through .dir-locals.el.
+
+If subprojects are Git submodules, you can use the variable
+`project-vc-merge-submodules' instead."
+  :type 'list
+  :version "28.1"
+  :safe (lambda (val) (and (listp val) (seq-every-p #'stringp val))))
+
 ;; FIXME: Using the current approach, major modes are supposed to set
 ;; this variable to a buffer-local value.  So we don't have access to
 ;; the "external roots" of language A from buffers of language B, which
@@ -428,7 +445,22 @@ project-try-vc
                       root)))))
             ('nil nil)
             (_ (ignore-errors (vc-call-backend backend 'root dir))))))
-    (and root (cons 'vc root))))
+    (when root
+      (let* ((relative-dir (file-relative-name dir root))
+             (subproject (seq-find
+                          (lambda (sub-dir)
+                            (string-prefix-p (file-name-as-directory sub-dir)
+                                             relative-dir))
+                          (project--value-in-dir
+                           'project-vc-subprojects
+                           dir))))
+        (if subproject
+            (cons 'vc (propertize
+                       (concat root subproject)
+                       ;; Side-channel so we don't change the value format.
+                       ;; But we could do that instead.
+                       'vc-subproject t))
+          (cons 'vc root))))))
 
 (defun project--submodule-p (root)
   ;; XXX: We only support Git submodules for now.
@@ -467,7 +499,9 @@ project-external-roots
 (cl-defmethod project-files ((project (head vc)) &optional dirs)
   (mapcan
    (lambda (dir)
-     (let ((ignores (project--value-in-dir 'project-vc-ignores dir))
+     (let ((ignores
+            (nconc (project--vc-subproject-ignores dir)
+                   (project--value-in-dir 'project-vc-ignores dir)))
            backend)
        (if (and (file-equal-p dir (cdr project))
                 (setq backend (vc-responsible-backend dir))
@@ -579,6 +613,12 @@ project--git-submodules
           (nreverse res)))
     (file-missing nil)))
 
+(defun project--vc-subproject-ignores (dir)
+  (unless (get-text-property 0 'vc-subproject dir)
+    (mapcar
+     (lambda (supb) (format "./%s" (file-name-as-directory supb)))
+     (project--value-in-dir 'project-vc-subprojects dir))))
+
 (cl-defmethod project-ignores ((project (head vc)) dir)
   (let* ((root (cdr project))
          backend)
@@ -608,6 +648,7 @@ project-ignores
              (vc-call-backend backend 'ignore-completion-table root)
            (vc-not-supported () nil)))))
      (project--value-in-dir 'project-vc-ignores root)
+     (project--vc-subproject-ignores root)
      (mapcar
       (lambda (dir)
         (concat dir "/"))

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

On 26/9/21 03:22, Dmitry Gutov wrote:
> Another issue is people working on monorepos (often backed by Git) 
> sometimes want to split them into separate projects. The latter 
> capability (excluding subproject's files) informed the choice of the 
> approach. Which is altering the vc project backend's behavior, rather 
> that offering this feature through another backend, like the one added 
> in the previous patch, for example.
> 
> I don't know if all users of this feature will want them excluded, 
> though. The attached implementation does, and maybe another option could 
> be added to disable this.
> 
> Or we could drop this part of the behavior, insisting that users who 
> want it could add the corresponding entries to project-vc-ignores. This 
> way they would be listing the subprojects twice, however. And the 
> project-vc-merge-submodules=nil behavior matches the other option 
> (submodule files are excluded from the parent).

Here's a third approach, which now seems to me the most approachable.

It solves the semantic problems by having the new "plain" projects also 
belong to the same type (which is still called project-vc, but means 
something more now). The backend's variables are adhered to, and 'git 
ls-files' is used for file listing in Git-backend subprojects still.

The subprojects' files are not excluded from the parent project, but as 
long as the format of markers stays with wildcards, it at least remains 
feasible to do, though with some extra expense at runtime. We'll see if 
people really want this.

It will probably also improve performance over Tramp compared to the 
current.

Does this work for everybody?

[-- Attachment #2: project-vc-extra-root-markers.diff --]
[-- Type: text/x-patch, Size: 5187 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 71061e6139d..19424a7343d 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -66,6 +66,9 @@
 ;; files, but supports additions to the list using the user option
 ;; `project-vc-ignores' (usually through .dir-locals.el).
 ;;
+;; At this point the name might as well be an abbreviation for "VC and
+;; Etc", see the variable `project-vc-extra-root-markers'.
+;;
 ;; Utils:
 ;;
 ;; `project-combine-directories' and `project-subtract-directories',
@@ -411,6 +414,27 @@ project-vc-name
   :version "29.1"
   :safe #'stringp)
 
+;; Not using regexps because these wouldn't work in Git pathspecs, in
+;; case we decide we need to be able to list subprojects.
+(defcustom project-vc-extra-root-markers nil
+  "List of additional \"markers\" to signal project roots.
+
+A marker is either a base file name or a glob pattern for such.
+
+These will be used in addition to regular directory markers such
+as .git, .hg, and so on, dependent on the value of
+`vc-handled-backends'.  They are most useful when a VC project
+has subdirectories inside it that need to be considered as
+separate projects, but still use the parent's ignore rules and
+general behaviors.
+
+It can also be used for projects outside of VC repositories.
+Their behavior will still obey the relevant variables, such as
+`project-vc-ignores' or `project-vc-name'."
+  :type 'list
+  :version "29.1"
+  :safe (lambda (val) (and (listp val) (cl-every #'stringp val))))
+
 ;; FIXME: Using the current approach, major modes are supposed to set
 ;; this variable to a buffer-local value.  So we don't have access to
 ;; the "external roots" of language A from buffers of language B, which
@@ -447,28 +471,50 @@ project-vc-external-roots-function
 backend implementation of `project-external-roots'.")
 
 (defun project-try-vc (dir)
+  (defvar vc-svn-admin-directory)
+  (require 'vc-svn)
+  ;; FIXME: Learn to invalidate when the value of
+  ;; `project-vc-merge-submodules' or `project-vc-extra-root-markers'
+  ;; changes.
   (or (vc-file-getprop dir 'project-vc)
-      (let* ((backend (ignore-errors (vc-responsible-backend dir)))
+      (let* ((backend-markers-alist `((Git . ".git")
+                                      (Hg . ".hg")
+                                      (Bzr . ".bzr")
+                                      (SVN . ,vc-svn-admin-directory)
+                                      (DARCS . "_darcs")
+                                      (Fossil . ".fslckout")))
+             (backend-markers
+              (delete
+               nil
+               (mapcar
+                (lambda (b) (assoc-default b backend-markers-alist))
+                vc-handled-backends)))
+             (marker-re
+              (mapconcat
+               (lambda (m) (format "\\(%s\\)" (wildcard-to-regexp m)))
+               (append backend-markers project-vc-extra-root-markers)
+               "\\|"))
+             (locate-dominating-stop-dir-regexp
+              (or vc-ignore-dir-regexp locate-dominating-stop-dir-regexp))
+             last-matches
              (root
-              (pcase backend
-                ('Git
-                 ;; Don't stop at submodule boundary.
-                 (or (vc-file-getprop dir 'project-git-root)
-                     (let ((root (vc-call-backend backend 'root dir)))
-                       (vc-file-setprop
-                        dir 'project-git-root
-                        (if (and
-                             ;; FIXME: Invalidate the cache when the value
-                             ;; of this variable changes.
-                             project-vc-merge-submodules
-                             (project--submodule-p root))
-                            (let* ((parent (file-name-directory
-                                            (directory-file-name root))))
-                              (vc-call-backend backend 'root parent))
-                          root)))))
-                ('nil nil)
-                (_ (ignore-errors (vc-call-backend backend 'root dir)))))
+              (locate-dominating-file
+               dir
+               (lambda (d)
+                 (setq last-matches (directory-files d nil marker-re t 100)))))
+             (backend
+              (cl-find-if
+               (lambda (b)
+                 (member (assoc-default b backend-markers-alist)
+                         last-matches))
+               vc-handled-backends))
              project)
+        (when (and
+               (eq backend 'Git)
+               project-vc-merge-submodules
+               (project--submodule-p root))
+          (let* ((parent (file-name-directory (directory-file-name root))))
+            (setq root (vc-call-backend 'Git 'root parent))))
         (when root
           (setq project (list 'vc backend root))
           ;; FIXME: Cache for a shorter time.
@@ -626,7 +672,8 @@ project-ignores
   (let* ((root (nth 2 project))
          backend)
     (append
-     (when (file-equal-p dir root)
+     (when (and backend
+                (file-equal-p dir root))
        (setq backend (cadr project))
        (delq
         nil

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Cc: "Philip K." <philipk@posteo.net>, Rudi Schlatte <rudi@constantly.at>,
>  Augusto Stoffel <arstoffel@gmail.com>, Zhu Zihao <cjpeople2013@gmail.com>,
>  Theodor Thornhill <theo@thornhill.no>,
>  Daniel Martín <mardani29@yahoo.es>,
>  Eric Abrahamsen <eric@ericabrahamsen.net>,
>  João Távora <joaotavora@gmail.com>,
>  Manuel Uberti <manuel.uberti@inventati.org>, Juri Linkov <juri@linkov.net>,
>  Rudolf Adamkovič <salutis@me.com>
> Date: Sat, 26 Nov 2022 03:49:36 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Does this work for everybody?

Hard to say, because:

> +(defcustom project-vc-extra-root-markers nil
> +  "List of additional \"markers\" to signal project roots.
> +
> +A marker is either a base file name or a glob pattern for such.
> +
> +These will be used in addition to regular directory markers such
> +as .git, .hg, and so on, dependent on the value of
> +`vc-handled-backends'.

"These will be used" how?  This crucial information is sorely missing from
this description.  Likewise, how "markers" that are not files are used and
are useful?

>                         They are most useful when a VC project
> +has subdirectories inside it that need to be considered as
> +separate projects, but still use the parent's ignore rules and
> +general behaviors.

How are "markers" useful in this scenario?

> +It can also be used for projects outside of VC repositories.
> +Their behavior will still obey the relevant variables, such as
> +`project-vc-ignores' or `project-vc-name'."

And in this one?

IOW, please describe the main ideas of this approach instead of relying on
use immediately gleaning that from a patch with incomplete documentation.

TIA

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 26/11/22 09:47, Eli Zaretskii wrote:

>> +(defcustom project-vc-extra-root-markers nil
>> +  "List of additional \"markers\" to signal project roots.
>> +
>> +A marker is either a base file name or a glob pattern for such.
>> +
>> +These will be used in addition to regular directory markers such
>> +as .git, .hg, and so on, dependent on the value of
>> +`vc-handled-backends'.
> 
> "These will be used" how?  This crucial information is sorely missing from
> this description.  Likewise, how "markers" that are not files are used and
> are useful?

Not files, meaning, markers which are globs with wildcards?

>>                          They are most useful when a VC project
>> +has subdirectories inside it that need to be considered as
>> +separate projects, but still use the parent's ignore rules and
>> +general behaviors.
> 
> How are "markers" useful in this scenario?

As mentioned in e.g. 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54228#11, monorepos often 
contain subdirectories which one might want to handle separately.

Those subdirectories often come in standard structures, e.g. a frontend 
subdirectory might contain a file called "package.json", a backend 
subdirectory might contain "Gemfile" or "build.gradle", or perhaps 
"autogen.sh", and so on.

Adding those to the markers list will tag those subdirectories as 
projects on their own. People can also use the file names special to 
their particular organization instead of those above.

>> +It can also be used for projects outside of VC repositories.
>> +Their behavior will still obey the relevant variables, such as
>> +`project-vc-ignores' or `project-vc-name'."
> 
> And in this one?

This covers the use cases described in the first messages of this and 
the merged bug report:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#5
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54228#5

People would add ".project" or ".emacs-project" to 
project-vc-extra-root-markers and see things work.

Might have to restart Emacs, though, if they already called project 
commands in the given directory, because the current project info is cached.

> IOW, please describe the main ideas of this approach instead of relying on
> use immediately gleaning that from a patch with incomplete documentation.

If you have further questions, please ask.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Sat, 26 Nov 2022 15:22:36 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  salutis@me.com, joaotavora@gmail.com, manuel.uberti@inventati.org,
>  juri@linkov.net, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 26/11/22 09:47, Eli Zaretskii wrote:
> 
> >> +(defcustom project-vc-extra-root-markers nil
> >> +  "List of additional \"markers\" to signal project roots.
> >> +
> >> +A marker is either a base file name or a glob pattern for such.
> >> +
> >> +These will be used in addition to regular directory markers such
> >> +as .git, .hg, and so on, dependent on the value of
> >> +`vc-handled-backends'.
> > 
> > "These will be used" how?  This crucial information is sorely missing from
> > this description.  Likewise, how "markers" that are not files are used and
> > are useful?
> 
> Not files, meaning, markers which are globs with wildcards?
> 
> >>                          They are most useful when a VC project
> >> +has subdirectories inside it that need to be considered as
> >> +separate projects, but still use the parent's ignore rules and
> >> +general behaviors.
> > 
> > How are "markers" useful in this scenario?
> 
> As mentioned in e.g. 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54228#11, monorepos often 
> contain subdirectories which one might want to handle separately.
> 
> Those subdirectories often come in standard structures, e.g. a frontend 
> subdirectory might contain a file called "package.json", a backend 
> subdirectory might contain "Gemfile" or "build.gradle", or perhaps 
> "autogen.sh", and so on.
> 
> Adding those to the markers list will tag those subdirectories as 
> projects on their own. People can also use the file names special to 
> their particular organization instead of those above.
> 
> >> +It can also be used for projects outside of VC repositories.
> >> +Their behavior will still obey the relevant variables, such as
> >> +`project-vc-ignores' or `project-vc-name'."
> > 
> > And in this one?
> 
> This covers the use cases described in the first messages of this and 
> the merged bug report:
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#5
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54228#5
> 
> People would add ".project" or ".emacs-project" to 
> project-vc-extra-root-markers and see things work.
> 
> Might have to restart Emacs, though, if they already called project 
> commands in the given directory, because the current project info is cached.
> 
> > IOW, please describe the main ideas of this approach instead of relying on
> > use immediately gleaning that from a patch with incomplete documentation.
> 
> If you have further questions, please ask.

Thanks, but I meant for at least some of the above to be in the doc strings
and/or explained here in plain English.  Readers aren't supposed to search
bug reports for relevant information.  (And for me personally, this means I
won't be able to look up the relevant information before at least another
week or two, which is a pity.)

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 3614 bytes --]

On 26/11/22 16:27, Eli Zaretskii wrote:
>> Date: Sat, 26 Nov 2022 15:22:36 +0200
>> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>>   cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>>   salutis@me.com, joaotavora@gmail.com, manuel.uberti@inventati.org,
>>   juri@linkov.net, arstoffel@gmail.com, 41572@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 26/11/22 09:47, Eli Zaretskii wrote:
>>
>>>> +(defcustom project-vc-extra-root-markers nil
>>>> +  "List of additional \"markers\" to signal project roots.
>>>> +
>>>> +A marker is either a base file name or a glob pattern for such.
>>>> +
>>>> +These will be used in addition to regular directory markers such
>>>> +as .git, .hg, and so on, dependent on the value of
>>>> +`vc-handled-backends'.
>>>
>>> "These will be used" how?  This crucial information is sorely missing from
>>> this description.  Likewise, how "markers" that are not files are used and
>>> are useful?
>>
>> Not files, meaning, markers which are globs with wildcards?
>>
>>>>                           They are most useful when a VC project
>>>> +has subdirectories inside it that need to be considered as
>>>> +separate projects, but still use the parent's ignore rules and
>>>> +general behaviors.
>>>
>>> How are "markers" useful in this scenario?
>>
>> As mentioned in e.g.
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54228#11, monorepos often
>> contain subdirectories which one might want to handle separately.
>>
>> Those subdirectories often come in standard structures, e.g. a frontend
>> subdirectory might contain a file called "package.json", a backend
>> subdirectory might contain "Gemfile" or "build.gradle", or perhaps
>> "autogen.sh", and so on.
>>
>> Adding those to the markers list will tag those subdirectories as
>> projects on their own. People can also use the file names special to
>> their particular organization instead of those above.
>>
>>>> +It can also be used for projects outside of VC repositories.
>>>> +Their behavior will still obey the relevant variables, such as
>>>> +`project-vc-ignores' or `project-vc-name'."
>>>
>>> And in this one?
>>
>> This covers the use cases described in the first messages of this and
>> the merged bug report:
>>
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#5
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54228#5
>>
>> People would add ".project" or ".emacs-project" to
>> project-vc-extra-root-markers and see things work.
>>
>> Might have to restart Emacs, though, if they already called project
>> commands in the given directory, because the current project info is cached.
>>
>>> IOW, please describe the main ideas of this approach instead of relying on
>>> use immediately gleaning that from a patch with incomplete documentation.
>>
>> If you have further questions, please ask.
> 
> Thanks, but I meant for at least some of the above to be in the doc strings
> and/or explained here in plain English.  Readers aren't supposed to search
> bug reports for relevant information.  (And for me personally, this means I
> won't be able to look up the relevant information before at least another
> week or two, which is a pity.)

The links were meant to illustrate that most people Cc'd probably 
understand the context already (modulo some forgetting because time has 
passed).

And either way I'm probably too close to the problem to understand 
easily what is clear to the average user and what is not. That's why 
questions are always useful.

I've added the examples to the docstring and rephrased it a little. Does 
it read better now?

[-- Attachment #2: project-vc-extra-root-markers-v2.diff --]
[-- Type: text/x-patch, Size: 5269 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 5b8648031fb..b4d5b4b0c9e 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -66,6 +66,9 @@
 ;; files, but supports additions to the list using the user option
 ;; `project-vc-ignores' (usually through .dir-locals.el).
 ;;
+;; At this point the name might as well be an abbreviation for "VC and
+;; Etc", see the variable `project-vc-extra-root-markers'.
+;;
 ;; Utils:
 ;;
 ;; `project-combine-directories' and `project-subtract-directories',
@@ -411,6 +414,29 @@ project-vc-name
   :version "29.1"
   :safe #'stringp)
 
+;; Not using regexps because these wouldn't work in Git pathspecs, in
+;; case we decide we need to be able to list subprojects.
+(defcustom project-vc-extra-root-markers nil
+  "List of additional markers to signal project roots.
+
+A marker is either a base file name or a glob pattern for such.
+
+Example values: \".dir-locals.el\", \"package.json\", \"pom.xml\",
+\"requirements.txt\", \"Gemfile\", \"*.gemspec\", \"autogen.sh\".
+
+These will be used in addition to regular directory markers such
+as .git, .hg, and so on, dependent on the value of
+`vc-handled-backends'.  They are most useful when a VC project
+has subdirectories inside it that need to be considered as
+separate projects.  It can also be used for projects outside of
+VC repositories.
+
+Either way, their behavior will still obey the relevant
+variables, such as `project-vc-ignores' or `project-vc-name'."
+  :type 'list
+  :version "29.1"
+  :safe (lambda (val) (and (listp val) (cl-every #'stringp val))))
+
 ;; FIXME: Using the current approach, major modes are supposed to set
 ;; this variable to a buffer-local value.  So we don't have access to
 ;; the "external roots" of language A from buffers of language B, which
@@ -447,28 +473,50 @@ project-vc-external-roots-function
 backend implementation of `project-external-roots'.")
 
 (defun project-try-vc (dir)
+  (defvar vc-svn-admin-directory)
+  (require 'vc-svn)
+  ;; FIXME: Learn to invalidate when the value of
+  ;; `project-vc-merge-submodules' or `project-vc-extra-root-markers'
+  ;; changes.
   (or (vc-file-getprop dir 'project-vc)
-      (let* ((backend (ignore-errors (vc-responsible-backend dir)))
+      (let* ((backend-markers-alist `((Git . ".git")
+                                      (Hg . ".hg")
+                                      (Bzr . ".bzr")
+                                      (SVN . ,vc-svn-admin-directory)
+                                      (DARCS . "_darcs")
+                                      (Fossil . ".fslckout")))
+             (backend-markers
+              (delete
+               nil
+               (mapcar
+                (lambda (b) (assoc-default b backend-markers-alist))
+                vc-handled-backends)))
+             (marker-re
+              (mapconcat
+               (lambda (m) (format "\\(%s\\)" (wildcard-to-regexp m)))
+               (append backend-markers project-vc-extra-root-markers)
+               "\\|"))
+             (locate-dominating-stop-dir-regexp
+              (or vc-ignore-dir-regexp locate-dominating-stop-dir-regexp))
+             last-matches
              (root
-              (pcase backend
-                ('Git
-                 ;; Don't stop at submodule boundary.
-                 (or (vc-file-getprop dir 'project-git-root)
-                     (let ((root (vc-call-backend backend 'root dir)))
-                       (vc-file-setprop
-                        dir 'project-git-root
-                        (if (and
-                             ;; FIXME: Invalidate the cache when the value
-                             ;; of this variable changes.
-                             project-vc-merge-submodules
-                             (project--submodule-p root))
-                            (let* ((parent (file-name-directory
-                                            (directory-file-name root))))
-                              (vc-call-backend backend 'root parent))
-                          root)))))
-                ('nil nil)
-                (_ (ignore-errors (vc-call-backend backend 'root dir)))))
+              (locate-dominating-file
+               dir
+               (lambda (d)
+                 (setq last-matches (directory-files d nil marker-re t 100)))))
+             (backend
+              (cl-find-if
+               (lambda (b)
+                 (member (assoc-default b backend-markers-alist)
+                         last-matches))
+               vc-handled-backends))
              project)
+        (when (and
+               (eq backend 'Git)
+               project-vc-merge-submodules
+               (project--submodule-p root))
+          (let* ((parent (file-name-directory (directory-file-name root))))
+            (setq root (vc-call-backend 'Git 'root parent))))
         (when root
           (setq project (list 'vc backend root))
           ;; FIXME: Cache for a shorter time.
@@ -626,7 +674,8 @@ project-ignores
   (let* ((root (nth 2 project))
          backend)
     (append
-     (when (file-equal-p dir root)
+     (when (and backend
+                (file-equal-p dir root))
        (setq backend (cadr project))
        (delq
         nil

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Sun, 27 Nov 2022 03:08:04 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  salutis@me.com, joaotavora@gmail.com, manuel.uberti@inventati.org,
>  juri@linkov.net, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >>> "These will be used" how?  This crucial information is sorely missing from
> >>> this description.  Likewise, how "markers" that are not files are used and
> >>> are useful?
> >>
> > [...]
> > Thanks, but I meant for at least some of the above to be in the doc strings
> > and/or explained here in plain English.  Readers aren't supposed to search
> > bug reports for relevant information.  (And for me personally, this means I
> > won't be able to look up the relevant information before at least another
> > week or two, which is a pity.)
> 
> The links were meant to illustrate that most people Cc'd probably 
> understand the context already (modulo some forgetting because time has 
> passed).
> 
> And either way I'm probably too close to the problem to understand 
> easily what is clear to the average user and what is not. That's why 
> questions are always useful.
> 
> I've added the examples to the docstring and rephrased it a little. Does 
> it read better now?

Sorry, no.  My basic question "how are these markers used?" is still
unanswered.  You seem to assume that saying "in addition to regular
directory markers such as .git, .hg" explains it, but it doesn't, because
how the "regular directory markers" are used is still a mystery.  And the
purpose of the markers according to the doc string, viz.:

   List of additional markers to signal project roots.

doesn't help enough, since "signal project roots" is too vague and abstract.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 27/11/22 08:46, Eli Zaretskii wrote:
> Sorry, no.  My basic question "how are these markers used?" is still
> unanswered.  You seem to assume that saying "in addition to regular
> directory markers such as .git, .hg" explains it, but it doesn't, because
> how the "regular directory markers" are used is still a mystery.  And the
> purpose of the markers according to the doc string, viz.:
> 
>     List of additional markers to signal project roots.
> 
> doesn't help enough, since "signal project roots" is too vague and abstract.

How about this, then? To borrow the phrasing from the very first patch 
in this bug:

   "List of additional markers to signal project roots.

A marker is either a base file name or a glob pattern for such.

A drectory containing such file or a file matching the pattern
will be recognized as a VC project.

Example values: \".dir-locals.el\", \"package.json\", \"pom.xml\",
\"requirements.txt\", \"Gemfile\", \"*.gemspec\", \"autogen.sh\".

These will be used in addition to regular directory markers such
as \".git\", \".hg\", and so on, depending on the value of
`vc-handled-backends'.  It is most useful when a VC project has
subdirectories inside it that need to be considered as separate
projects.  It can also be used for projects outside of VC
repositories.

In either case, their behavior will still obey the relevant
variables, such as `project-vc-ignores' or `project-vc-name'."

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Sun, 27 Nov 2022 16:10:02 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  salutis@me.com, joaotavora@gmail.com, manuel.uberti@inventati.org,
>  juri@linkov.net, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 27/11/22 08:46, Eli Zaretskii wrote:
> > Sorry, no.  My basic question "how are these markers used?" is still
> > unanswered.  You seem to assume that saying "in addition to regular
> > directory markers such as .git, .hg" explains it, but it doesn't, because
> > how the "regular directory markers" are used is still a mystery.  And the
> > purpose of the markers according to the doc string, viz.:
> > 
> >     List of additional markers to signal project roots.
> > 
> > doesn't help enough, since "signal project roots" is too vague and abstract.
> 
> How about this, then? To borrow the phrasing from the very first patch 
> in this bug:
> 
>    "List of additional markers to signal project roots.
> 
> A marker is either a base file name or a glob pattern for such.
> 
> A drectory containing such file or a file matching the pattern
> will be recognized as a VC project.

Better, but the last sentence above should say

  A directory containing such a marker file or a file matching a marker
  pattern will be recognized as the root of a VC project.

(Btw, why "VC project"? can't I use marker files for non-VC projects?)

Thanks.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 27/11/22 16:27, Eli Zaretskii wrote:
> Better, but the last sentence above should say
> 
>    A directory containing such a marker file or a file matching a marker
>    pattern will be recognized as the root of a VC project.

Okay.

> (Btw, why "VC project"? can't I use marker files for non-VC projects?)

Yes, you can. That's what the docstring says: "can also be used for 
projects outside of VC repositories".

But "VC project" is a proper noun in this usage. Basically, a "VC 
project" is whatever value (if non-nil) that is returned by project-try-vc.

It's meaningful to have some proper noun for this, because this project 
type has customization variables, and it's handy to be able to use them 
for projects outside of VC repositories as well, recognized according to 
the new option.

Like the patch also says (and what's given me a pause in the past), that 
also makes "VC project" somewhat a misnomer. But I'm not sure what to 
call them better, and renaming a whole bunch of symbols creates a 
backward incompatibility after all. Even though, luckily, we've asked 
people not to rely on that object's internals.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Sun, 27 Nov 2022 17:51:45 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>  salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 27/11/22 16:27, Eli Zaretskii wrote:
> 
> > (Btw, why "VC project"? can't I use marker files for non-VC projects?)
> 
> Yes, you can. That's what the docstring says: "can also be used for 
> projects outside of VC repositories".
> 
> But "VC project" is a proper noun in this usage. Basically, a "VC 
> project" is whatever value (if non-nil) that is returned by project-try-vc.

Then maybe change that to something like

    A directory containing such a marker file or a file matching a marker
    pattern will be recognized as the root of a project whose type is
    "VC project".

The point of this is to tell that those markers indicate projects whose type
just happens to be "VC project".  Otherwise the above could be interpreted
that the markers can be used only inside VC projects.

> Like the patch also says (and what's given me a pause in the past), that 
> also makes "VC project" somewhat a misnomer. But I'm not sure what to 
> call them better

How about VC-backed project?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 27/11/22 18:43, Eli Zaretskii wrote:
>> Date: Sun, 27 Nov 2022 17:51:45 +0200
>> Cc:philipk@posteo.net,rudi@constantly.at,eric@ericabrahamsen.net,
>>   cjpeople2013@gmail.com,theo@thornhill.no,mardani29@yahoo.es,
>>   joaotavora@gmail.com,manuel.uberti@inventati.org,juri@linkov.net,
>>   salutis@me.com,arstoffel@gmail.com,41572@debbugs.gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 27/11/22 16:27, Eli Zaretskii wrote:
>>
>>> (Btw, why "VC project"? can't I use marker files for non-VC projects?)
>> Yes, you can. That's what the docstring says: "can also be used for
>> projects outside of VC repositories".
>>
>> But "VC project" is a proper noun in this usage. Basically, a "VC
>> project" is whatever value (if non-nil) that is returned by project-try-vc.
> Then maybe change that to something like
> 
>      A directory containing such a marker file or a file matching a marker
>      pattern will be recognized as the root of a project whose type is
>      "VC project".

If you say quote help, it's fine by me.

> The point of this is to tell that those markers indicate projects whose type
> just happens to be "VC project".  Otherwise the above could be interpreted
> that the markers can be used only inside VC projects.

Sure.

>> Like the patch also says (and what's given me a pause in the past), that
>> also makes "VC project" somewhat a misnomer. But I'm not sure what to
>> call them better
> How about VC-backed project?

Is that different? I would say it might be worse: "VC-backed" sounds 
like a project that must correspond to a VC repository (be "backed" by it).

OTOH a new term we could use would stand for something like: a project 
type which will recognize the enabled kinds of VC repositories and use 
their roots as project root, and knows how to use certain VC systems to 
speed up the fetching of files, and knows about Git submodules, but also 
recognizes other directories (inside or outside of VC repositories) 
based on configurable conditions.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Sun, 27 Nov 2022 23:41:59 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>  salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >      A directory containing such a marker file or a file matching a marker
> >      pattern will be recognized as the root of a project whose type is
> >      "VC project".
> 
> If you say quote help, it's fine by me.
> 
> > The point of this is to tell that those markers indicate projects whose type
> > just happens to be "VC project".  Otherwise the above could be interpreted
> > that the markers can be used only inside VC projects.
> 
> Sure.
> 
> >> Like the patch also says (and what's given me a pause in the past), that
> >> also makes "VC project" somewhat a misnomer. But I'm not sure what to
> >> call them better
> > How about VC-backed project?
> 
> Is that different?

I don't know.  You asked for alternatives, and that is what I could think
of.

> I would say it might be worse: "VC-backed" sounds 
> like a project that must correspond to a VC repository (be "backed" by it).

It usually is, isn't it?

If it doesn't have to, then we should look for a different name ASAP, one
that doesn't include "VC" at all.

> OTOH a new term we could use would stand for something like: a project 
> type which will recognize the enabled kinds of VC repositories and use 
> their roots as project root, and knows how to use certain VC systems to 
> speed up the fetching of files, and knows about Git submodules, but also 
> recognizes other directories (inside or outside of VC repositories) 
> based on configurable conditions.

If this type of project doesn't have to be "backed by a VCS", then what does
distinguish it from other types of projects?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 28/11/2022 13:58, Eli Zaretskii wrote:

>>>> Like the patch also says (and what's given me a pause in the past), that
>>>> also makes "VC project" somewhat a misnomer. But I'm not sure what to
>>>> call them better
>>> How about VC-backed project?
>>
>> Is that different?
> 
> I don't know.  You asked for alternatives, and that is what I could think
> of.
> 
>> I would say it might be worse: "VC-backed" sounds
>> like a project that must correspond to a VC repository (be "backed" by it).
> 
> It usually is, isn't it?

Usually -- yes, the default ootb configuration will have projects backed 
by VC repositories, because the default set of markers is determined by 
the value of 'vc-handled-backends'.

But now the users will be able to extend that set of markers through 
customization.

> If it doesn't have to, then we should look for a different name ASAP, one
> that doesn't include "VC" at all.

I haven't been able to come up with anything more concrete than 
"builtin", and that's both a non-name and not exactly true since 
'transient' also exists. But "VC aware" might also fit the bill. It 
sounds more like an adjective, though, than a proper noun.

>> OTOH a new term we could use would stand for something like: a project
>> type which will recognize the enabled kinds of VC repositories and use
>> their roots as project root, and knows how to use certain VC systems to
>> speed up the fetching of files, and knows about Git submodules, but also
>> recognizes other directories (inside or outside of VC repositories)
>> based on configurable conditions.
> 
> If this type of project doesn't have to be "backed by a VCS", then what does
> distinguish it from other types of projects?

It difficult to come up with a name using a principle like that when the 
preceding effort was made to be able to accommodate the projects of 
different users with new usage patterns.

It is "VC aware" or maybe "VC friendly", and it works with projects in 
the shape of directory trees. But both characterizations can apply to 
e.g. Projectile as well. Except the latter is probably more flexible, 
and our backend is probably faster, especially over Tramp.

But if we just compare with the other builtin ('transient'), then its 
VC awareness is probably the key thing. And the customization options it 
has (which 'transient' doesn't).

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Mon, 28 Nov 2022 14:30:15 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>  salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > If this type of project doesn't have to be "backed by a VCS", then what does
> > distinguish it from other types of projects?
> 
> It difficult to come up with a name using a principle like that when the 
> preceding effort was made to be able to accommodate the projects of 
> different users with new usage patterns.
> 
> It is "VC aware" or maybe "VC friendly", and it works with projects in 
> the shape of directory trees. But both characterizations can apply to 
> e.g. Projectile as well. Except the latter is probably more flexible, 
> and our backend is probably faster, especially over Tramp.
> 
> But if we just compare with the other builtin ('transient'), then its 
> VC awareness is probably the key thing. And the customization options it 
> has (which 'transient' doesn't).

AFAIU, the 'transient' builtin also works with projects whose shape is a
directory tree, right?  If so, how is 'transient' different from the
'VC-aware' type?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 28/11/2022 15:22, Eli Zaretskii wrote:
>> Date: Mon, 28 Nov 2022 14:30:15 +0200
>> Cc:philipk@posteo.net,rudi@constantly.at,eric@ericabrahamsen.net,
>>   cjpeople2013@gmail.com,theo@thornhill.no,mardani29@yahoo.es,
>>   joaotavora@gmail.com,manuel.uberti@inventati.org,juri@linkov.net,
>>   salutis@me.com,arstoffel@gmail.com,41572@debbugs.gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>> If this type of project doesn't have to be "backed by a VCS", then what does
>>> distinguish it from other types of projects?
>> It difficult to come up with a name using a principle like that when the
>> preceding effort was made to be able to accommodate the projects of
>> different users with new usage patterns.
>>
>> It is "VC aware" or maybe "VC friendly", and it works with projects in
>> the shape of directory trees. But both characterizations can apply to
>> e.g. Projectile as well. Except the latter is probably more flexible,
>> and our backend is probably faster, especially over Tramp.
>>
>> But if we just compare with the other builtin ('transient'), then its
>> VC awareness is probably the key thing. And the customization options it
>> has (which 'transient' doesn't).
> AFAIU, the 'transient' builtin also works with projects whose shape is a
> directory tree, right?  If so, how is 'transient' different from the
> 'VC-aware' type?

That's right.

So if we are comparing against 'transient' only, then VC awareness is 
probably the key thing. And the customization options.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Mon, 28 Nov 2022 16:29:15 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>  salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> But if we just compare with the other builtin ('transient'), then its
> >> VC awareness is probably the key thing. And the customization options it
> >> has (which 'transient' doesn't).
> > AFAIU, the 'transient' builtin also works with projects whose shape is a
> > directory tree, right?  If so, how is 'transient' different from the
> > 'VC-aware' type?
> 
> That's right.

So you are saying that 'VC-aware' is like 'transient', but it will use a VCS
if it's available?  Then I guess 'VC-aware' is a good name for it.

> So if we are comparing against 'transient' only, then VC awareness is 
> probably the key thing. And the customization options.

Do we have other types to compare against?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 28/11/2022 16:49, Eli Zaretskii wrote:
>> Date: Mon, 28 Nov 2022 16:29:15 +0200
>> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>>   cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>>   joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>>   salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> But if we just compare with the other builtin ('transient'), then its
>>>> VC awareness is probably the key thing. And the customization options it
>>>> has (which 'transient' doesn't).
>>> AFAIU, the 'transient' builtin also works with projects whose shape is a
>>> directory tree, right?  If so, how is 'transient' different from the
>>> 'VC-aware' type?
>>
>> That's right.
> 
> So you are saying that 'VC-aware' is like 'transient', but it will use a VCS
> if it's available?  Then I guess 'VC-aware' is a good name for it.

I suppose so. Great!

>> So if we are comparing against 'transient' only, then VC awareness is
>> probably the key thing. And the customization options.
> 
> Do we have other types to compare against?

I mentioned Projectile a few messages ago.

It's a popular alternative that, with the last release, also plugs into 
project-find-functions and thus works as a new project.el backend.

But it's third-party.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Mon, 28 Nov 2022 17:31:11 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>  salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > So you are saying that 'VC-aware' is like 'transient', but it will use a VCS
> > if it's available?  Then I guess 'VC-aware' is a good name for it.
> 
> I suppose so. Great!
> 
> >> So if we are comparing against 'transient' only, then VC awareness is
> >> probably the key thing. And the customization options.
> > 
> > Do we have other types to compare against?
> 
> I mentioned Projectile a few messages ago.
> 
> It's a popular alternative that, with the last release, also plugs into 
> project-find-functions and thus works as a new project.el backend.
> 
> But it's third-party.

Well, we can only compare to objects we manage ourselves, so Projectile is
not relevant.

Then okay, we have 2 built-in project types, and the difference is that one
will use a VCS when available, the other won't.  That's clear enough to have
in the docs, I think.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 28/11/2022 18:51, Eli Zaretskii wrote:
> Then okay, we have 2 built-in project types, and the difference is that one
> will use a VCS when available, the other won't.  That's clear enough to have
> in the docs, I think.

Very good.

Eli, what do you think about this feature 
(project-vc-extra-root-markers) for emacs-29?

This bug has been open a while. I'm not seeing much user feedback now, 
but that's probably because everyone has been living with their own 
custom solution for all this time. And we'll still have time for people 
to report any bugs before the release.

Further, I've done some testing over Tramp, and the new approach 
improves the performance of the default implementation significantly. 
Like with a remote host over 200ms ping, with clear Tramp cache, "cold" 
project-current call is down from 3-6s to 1.9-3.3s (depending on the 
directory depth).

Not sure how many are working over such a slow connection, but that 
could be a nice bump.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 04:26:36 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>  salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 28/11/2022 18:51, Eli Zaretskii wrote:
> > Then okay, we have 2 built-in project types, and the difference is that one
> > will use a VCS when available, the other won't.  That's clear enough to have
> > in the docs, I think.
> 
> Very good.
> 
> Eli, what do you think about this feature 
> (project-vc-extra-root-markers) for emacs-29?

Where can I see the code that you are proposing?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 854 bytes --]

On 30/11/2022 15:29, Eli Zaretskii wrote:
>> Date: Wed, 30 Nov 2022 04:26:36 +0200
>> Cc:philipk@posteo.net,rudi@constantly.at,eric@ericabrahamsen.net,
>>   cjpeople2013@gmail.com,theo@thornhill.no,mardani29@yahoo.es,
>>   joaotavora@gmail.com,manuel.uberti@inventati.org,juri@linkov.net,
>>   salutis@me.com,arstoffel@gmail.com,41572@debbugs.gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 28/11/2022 18:51, Eli Zaretskii wrote:
>>> Then okay, we have 2 built-in project types, and the difference is that one
>>> will use a VCS when available, the other won't.  That's clear enough to have
>>> in the docs, I think.
>> Very good.
>>
>> Eli, what do you think about this feature
>> (project-vc-extra-root-markers) for emacs-29?
> Where can I see the code that you are proposing?

Here you go, I also added some documentation updates and 2 tests.

[-- Attachment #2: project-vc-extra-root-markers-v3.diff --]
[-- Type: text/x-patch, Size: 8383 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index cc28bddff22..af136ab60e9 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1,7 +1,7 @@
 ;;; project.el --- Operations on the current project  -*- lexical-binding: t; -*-
 
 ;; Copyright (C) 2015-2022 Free Software Foundation, Inc.
-;; Version: 0.8.3
+;; Version: 0.9.0
 ;; Package-Requires: ((emacs "26.1") (xref "1.4.0"))
 
 ;; This is a GNU ELPA :core package.  Avoid using functionality that
@@ -58,13 +58,30 @@
 ;;
 ;; This list can change in future versions.
 ;;
-;; VC project:
+;; Transient project:
+;;
+;; An instance of this type can be returned by `project-current' if no
+;; project was detected automatically, and the user had to pick a
+;; directory manually.  The fileset it describes is the whole
+;; directory, with the exception of some standard ignored files and
+;; directories.  This type has little purpose otherwise, as the only
+;; generic function it provides an override for is `project-root'.
+;;
+;; VC-aware project:
 ;;
 ;; Originally conceived as an example implementation, now it's a
 ;; relatively fast backend that delegates to 'git ls-files' or 'hg
 ;; status' to list the project's files.  It honors the VC ignore
 ;; files, but supports additions to the list using the user option
-;; `project-vc-ignores' (usually through .dir-locals.el).
+;; `project-vc-ignores' (usually through .dir-locals.el).  See the
+;; customization group `project-vc' for other options that control its
+;; behavior.
+;;
+;; If the repository is using any other VCS than Git or Hg, the file
+;; listing uses the default mechanism based on 'find'.
+;;
+;; This project type can also be used for non-VCS controlled
+;; directories, see the variable `project-vc-extra-root-markers'.
 ;;
 ;; Utils:
 ;;
@@ -377,7 +394,7 @@ project-buffers
     (nreverse bufs)))
 
 (defgroup project-vc nil
-  "Project implementation based on the VC package."
+  "VC-aware project implementation."
   :version "25.1"
   :group 'project)
 
@@ -397,21 +414,48 @@ project-vc-merge-submodules
   :safe #'booleanp)
 
 (defcustom project-vc-include-untracked t
-  "When non-nil, the VC project backend includes untracked files."
+  "When non-nil, the VC-aware project backend includes untracked files."
   :type 'boolean
   :version "29.1"
   :safe #'booleanp)
 
 (defcustom project-vc-name nil
-  "When non-nil, the name of the current VC project.
+  "When non-nil, the name of the current VC-aware project.
 
-The best way to change the value a VC project reports as its
-name, is by setting this in .dir-locals.el."
+The best way to change the value a VC-aware project reports as
+its name, is by setting this in .dir-locals.el."
   :type '(choice (const :tag "Default to the base name" nil)
                  (string :tag "Custom name"))
   :version "29.1"
   :safe #'stringp)
 
+;; Not using regexps because these wouldn't work in Git pathspecs, in
+;; case we decide we need to be able to list nested projects.
+(defcustom project-vc-extra-root-markers nil
+  "List of additional markers to signal project roots.
+
+A marker is either a base file name or a glob pattern for such.
+
+A directory containing such a marker file or a file matching a
+marker pattern will be recognized as the root of a VC-aware
+project.
+
+Example values: \".dir-locals.el\", \"package.json\", \"pom.xml\",
+\"requirements.txt\", \"Gemfile\", \"*.gemspec\", \"autogen.sh\".
+
+These will be used in addition to regular directory markers such
+as \".git\", \".hg\", and so on, depending on the value of
+`vc-handled-backends'.  It is most useful when a project has
+subdirectories inside it that need to be considered as separate
+projects.  It can also be used for projects outside of VC
+repositories.
+
+In either case, their behavior will still obey the relevant
+variables, such as `project-vc-ignores' or `project-vc-name'."
+  :type 'list
+  :version "29.1"
+  :safe (lambda (val) (and (listp val) (cl-every #'stringp val))))
+
 ;; FIXME: Using the current approach, major modes are supposed to set
 ;; this variable to a buffer-local value.  So we don't have access to
 ;; the "external roots" of language A from buffers of language B, which
@@ -420,7 +464,7 @@ project-vc-name
 ;;
 ;; We could add a second argument to this function: a file extension,
 ;; or a language name.  Some projects will know the set of languages
-;; used in them; for others, like VC-based projects, we'll need
+;; used in them; for others, like the VC-aware type, we'll need
 ;; auto-detection.  I see two options:
 ;;
 ;; - That could be implemented as a separate second hook, with a
@@ -444,32 +488,54 @@ project-vc-external-roots-function
 It should return a list of directory roots that contain source
 files related to the current buffer.
 
-The directory names should be absolute.  Used in the VC project
-backend implementation of `project-external-roots'.")
+The directory names should be absolute.  Used in the VC-aware
+project backend implementation of `project-external-roots'.")
 
 (defun project-try-vc (dir)
+  (defvar vc-svn-admin-directory)
+  (require 'vc-svn)
+  ;; FIXME: Learn to invalidate when the value of
+  ;; `project-vc-merge-submodules' or `project-vc-extra-root-markers'
+  ;; changes.
   (or (vc-file-getprop dir 'project-vc)
-      (let* ((backend (ignore-errors (vc-responsible-backend dir)))
+      (let* ((backend-markers-alist `((Git . ".git")
+                                      (Hg . ".hg")
+                                      (Bzr . ".bzr")
+                                      (SVN . ,vc-svn-admin-directory)
+                                      (DARCS . "_darcs")
+                                      (Fossil . ".fslckout")))
+             (backend-markers
+              (delete
+               nil
+               (mapcar
+                (lambda (b) (assoc-default b backend-markers-alist))
+                vc-handled-backends)))
+             (marker-re
+              (mapconcat
+               (lambda (m) (format "\\(%s\\)" (wildcard-to-regexp m)))
+               (append backend-markers project-vc-extra-root-markers)
+               "\\|"))
+             (locate-dominating-stop-dir-regexp
+              (or vc-ignore-dir-regexp locate-dominating-stop-dir-regexp))
+             last-matches
              (root
-              (pcase backend
-                ('Git
-                 ;; Don't stop at submodule boundary.
-                 (or (vc-file-getprop dir 'project-git-root)
-                     (let ((root (vc-call-backend backend 'root dir)))
-                       (vc-file-setprop
-                        dir 'project-git-root
-                        (if (and
-                             ;; FIXME: Invalidate the cache when the value
-                             ;; of this variable changes.
-                             project-vc-merge-submodules
-                             (project--submodule-p root))
-                            (let* ((parent (file-name-directory
-                                            (directory-file-name root))))
-                              (vc-call-backend backend 'root parent))
-                          root)))))
-                ('nil nil)
-                (_ (ignore-errors (vc-call-backend backend 'root dir)))))
+              (locate-dominating-file
+               dir
+               (lambda (d)
+                 (setq last-matches (directory-files d nil marker-re t 100)))))
+             (backend
+              (cl-find-if
+               (lambda (b)
+                 (member (assoc-default b backend-markers-alist)
+                         last-matches))
+               vc-handled-backends))
              project)
+        (when (and
+               (eq backend 'Git)
+               project-vc-merge-submodules
+               (project--submodule-p root))
+          (let* ((parent (file-name-directory (directory-file-name root))))
+            (setq root (vc-call-backend 'Git 'root parent))))
         (when root
           (setq project (list 'vc backend root))
           ;; FIXME: Cache for a shorter time.
@@ -627,7 +693,8 @@ project-ignores
   (let* ((root (nth 2 project))
          backend)
     (append
-     (when (file-equal-p dir root)
+     (when (and backend
+                (file-equal-p dir root))
        (setq backend (cadr project))
        (delq
         nil

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 20:52:32 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>  salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> Eli, what do you think about this feature
> >> (project-vc-extra-root-markers) for emacs-29?
> > Where can I see the code that you are proposing?
> 
> Here you go, I also added some documentation updates and 2 tests.

Thanks.  But I don't see any tests...

> +;; If the repository is using any other VCS than Git or Hg, the file
> +;; listing uses the default mechanism based on 'find'.

Instead of a literal 'find', this should probably say something like

  If the repository is using any other VCS than Git or Hg, the file
  listing uses the default mechanism based on the program specified by
  `find-program'.

>  (defun project-try-vc (dir)
> +  (defvar vc-svn-admin-directory)
> +  (require 'vc-svn)
> +  ;; FIXME: Learn to invalidate when the value of
> +  ;; `project-vc-merge-submodules' or `project-vc-extra-root-markers'
> +  ;; changes.
>    (or (vc-file-getprop dir 'project-vc)
> -      (let* ((backend (ignore-errors (vc-responsible-backend dir)))
> +      (let* ((backend-markers-alist `((Git . ".git")
> +                                      (Hg . ".hg")
> +                                      (Bzr . ".bzr")
> +                                      (SVN . ,vc-svn-admin-directory)
> +                                      (DARCS . "_darcs")
> +                                      (Fossil . ".fslckout")))
> +             (backend-markers
> +              (delete
> +               nil
> +               (mapcar
> +                (lambda (b) (assoc-default b backend-markers-alist))
> +                vc-handled-backends)))
> +             (marker-re
> +              (mapconcat
> +               (lambda (m) (format "\\(%s\\)" (wildcard-to-regexp m)))
> +               (append backend-markers project-vc-extra-root-markers)
> +               "\\|"))
> +             (locate-dominating-stop-dir-regexp
> +              (or vc-ignore-dir-regexp locate-dominating-stop-dir-regexp))
> +             last-matches
>               (root
> -              (pcase backend
> -                ('Git
> -                 ;; Don't stop at submodule boundary.
> -                 (or (vc-file-getprop dir 'project-git-root)
> -                     (let ((root (vc-call-backend backend 'root dir)))
> -                       (vc-file-setprop
> -                        dir 'project-git-root
> -                        (if (and
> -                             ;; FIXME: Invalidate the cache when the value
> -                             ;; of this variable changes.
> -                             project-vc-merge-submodules
> -                             (project--submodule-p root))
> -                            (let* ((parent (file-name-directory
> -                                            (directory-file-name root))))
> -                              (vc-call-backend backend 'root parent))
> -                          root)))))
> -                ('nil nil)
> -                (_ (ignore-errors (vc-call-backend backend 'root dir)))))
> +              (locate-dominating-file
> +               dir
> +               (lambda (d)
> +                 (setq last-matches (directory-files d nil marker-re t 100)))))
> +             (backend
> +              (cl-find-if
> +               (lambda (b)
> +                 (member (assoc-default b backend-markers-alist)
> +                         last-matches))
> +               vc-handled-backends))
>               project)
> +        (when (and
> +               (eq backend 'Git)
> +               project-vc-merge-submodules
> +               (project--submodule-p root))
> +          (let* ((parent (file-name-directory (directory-file-name root))))
> +            (setq root (vc-call-backend 'Git 'root parent))))
>          (when root
>            (setq project (list 'vc backend root))
>            ;; FIXME: Cache for a shorter time.

This is a significant change of the implementation of a public API.  Isn't
it risky to make such changes on the release branch?

But if you are okay with that, it's fine by me.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 30/11/2022 22:32, Eli Zaretskii wrote:
>> Date: Wed, 30 Nov 2022 20:52:32 +0200
>> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>>   cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>>   joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>>   salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> Eli, what do you think about this feature
>>>> (project-vc-extra-root-markers) for emacs-29?
>>> Where can I see the code that you are proposing?
>>
>> Here you go, I also added some documentation updates and 2 tests.
> 
> Thanks.  But I don't see any tests...

Sorry, missed them in this patch. They don't really need an advance 
review, though, so just see them later.

>> +;; If the repository is using any other VCS than Git or Hg, the file
>> +;; listing uses the default mechanism based on 'find'.
> 
> Instead of a literal 'find', this should probably say something like
> 
>    If the repository is using any other VCS than Git or Hg, the file
>    listing uses the default mechanism based on the program specified by
>    `find-program'.

Sure, thanks.

>>   (defun project-try-vc (dir)
>> +  (defvar vc-svn-admin-directory)
>> +  (require 'vc-svn)
>> +  ;; FIXME: Learn to invalidate when the value of
>> +  ;; `project-vc-merge-submodules' or `project-vc-extra-root-markers'
>> +  ;; changes.
>>     (or (vc-file-getprop dir 'project-vc)
>> -      (let* ((backend (ignore-errors (vc-responsible-backend dir)))
>> +      (let* ((backend-markers-alist `((Git . ".git")
>> +                                      (Hg . ".hg")
>> +                                      (Bzr . ".bzr")
>> +                                      (SVN . ,vc-svn-admin-directory)
>> +                                      (DARCS . "_darcs")
>> +                                      (Fossil . ".fslckout")))
>> +             (backend-markers
>> +              (delete
>> +               nil
>> +               (mapcar
>> +                (lambda (b) (assoc-default b backend-markers-alist))
>> +                vc-handled-backends)))
>> +             (marker-re
>> +              (mapconcat
>> +               (lambda (m) (format "\\(%s\\)" (wildcard-to-regexp m)))
>> +               (append backend-markers project-vc-extra-root-markers)
>> +               "\\|"))
>> +             (locate-dominating-stop-dir-regexp
>> +              (or vc-ignore-dir-regexp locate-dominating-stop-dir-regexp))
>> +             last-matches
>>                (root
>> -              (pcase backend
>> -                ('Git
>> -                 ;; Don't stop at submodule boundary.
>> -                 (or (vc-file-getprop dir 'project-git-root)
>> -                     (let ((root (vc-call-backend backend 'root dir)))
>> -                       (vc-file-setprop
>> -                        dir 'project-git-root
>> -                        (if (and
>> -                             ;; FIXME: Invalidate the cache when the value
>> -                             ;; of this variable changes.
>> -                             project-vc-merge-submodules
>> -                             (project--submodule-p root))
>> -                            (let* ((parent (file-name-directory
>> -                                            (directory-file-name root))))
>> -                              (vc-call-backend backend 'root parent))
>> -                          root)))))
>> -                ('nil nil)
>> -                (_ (ignore-errors (vc-call-backend backend 'root dir)))))
>> +              (locate-dominating-file
>> +               dir
>> +               (lambda (d)
>> +                 (setq last-matches (directory-files d nil marker-re t 100)))))
>> +             (backend
>> +              (cl-find-if
>> +               (lambda (b)
>> +                 (member (assoc-default b backend-markers-alist)
>> +                         last-matches))
>> +               vc-handled-backends))
>>                project)
>> +        (when (and
>> +               (eq backend 'Git)
>> +               project-vc-merge-submodules
>> +               (project--submodule-p root))
>> +          (let* ((parent (file-name-directory (directory-file-name root))))
>> +            (setq root (vc-call-backend 'Git 'root parent))))
>>           (when root
>>             (setq project (list 'vc backend root))
>>             ;; FIXME: Cache for a shorter time.
> 
> This is a significant change of the implementation of a public API.  Isn't
> it risky to make such changes on the release branch?
> 
> But if you are okay with that, it's fine by me.

A little bit, yeah. But I've done some dogfooding, and we have a couple 
of months before the release, right?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

Version: 29.1

On 30/11/2022 22:43, Dmitry Gutov wrote:
>>
>> This is a significant change of the implementation of a public API.  
>> Isn't
>> it risky to make such changes on the release branch?
>>
>> But if you are okay with that, it's fine by me.
> 
> A little bit, yeah. But I've done some dogfooding, and we have a couple 
> of months before the release, right?

And now pushed to emacs-29.

Looks like time to close this bug (and the merged one). The new user 
option project-vc-extra-root-markers should cover the described use 
cases in both feature requests. The default is nil, though, so people 
will need to customize.

Not every idea from this discussion has made it in, but we can get back 
to them later.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 22:43:59 +0200
> Cc: philipk@posteo.net, rudi@constantly.at, eric@ericabrahamsen.net,
>  cjpeople2013@gmail.com, theo@thornhill.no, mardani29@yahoo.es,
>  joaotavora@gmail.com, manuel.uberti@inventati.org, juri@linkov.net,
>  salutis@me.com, arstoffel@gmail.com, 41572@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > This is a significant change of the implementation of a public API.  Isn't
> > it risky to make such changes on the release branch?
> > 
> > But if you are okay with that, it's fine by me.
> 
> A little bit, yeah. But I've done some dogfooding, and we have a couple 
> of months before the release, right?

Yes.  The question is, how much will this be used till then.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 01/12/2022 08:02, Eli Zaretskii wrote:
>> Date: Wed, 30 Nov 2022 22:43:59 +0200
>> Cc:philipk@posteo.net,rudi@constantly.at,eric@ericabrahamsen.net,
>>   cjpeople2013@gmail.com,theo@thornhill.no,mardani29@yahoo.es,
>>   joaotavora@gmail.com,manuel.uberti@inventati.org,juri@linkov.net,
>>   salutis@me.com,arstoffel@gmail.com,41572@debbugs.gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>> This is a significant change of the implementation of a public API.  Isn't
>>> it risky to make such changes on the release branch?
>>>
>>> But if you are okay with that, it's fine by me.
>> A little bit, yeah. But I've done some dogfooding, and we have a couple
>> of months before the release, right?
> Yes.  The question is, how much will this be used till then.

True enough.

I've updated the docs and NEWS in the meantime, this should help.

On the bright side, though, the new feature uses the same code path as 
the core functionality, so it's still indirectly tested by anyone using 
the built-in project support.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Does this work for everybody?

I was pointed to this thread and asked to comment on this patch.

My use case is the following: I'm interested in being able to designate
projects (through various means, not only marker files) that may only
exist inside other projects.  I then want the C-x p family of commands
to allow a choice of inner project or any of the associated
super-projects.

I can't understand from the patch alone if it solves this case, but it
seems that it doesn't come near.  If not for any other reason, I can't
always use marker files.

I do see that in your patch more and more things appeari under the
existing VC type, which I think is growing too much.  The comment itself
admits that "VC" becomes "VC and etc." -- this is not a good sign.

The patch seems to be trying very hard not to create a new project type.

But if a new subproject type was created with a link to a previously
found super-project.  One could easily e.g. reuse the super-project's
ignore rules etc in the sub-project.

João

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 26/11/22 11:52, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Does this work for everybody?
> 
> I was pointed to this thread and asked to comment on this patch.
> 
> My use case is the following: I'm interested in being able to designate
> projects (through various means, not only marker files) that may only
> exist inside other projects.

You previously described your super-project and how you handled it using 
project-find-functions hook with a new element that looked for file 
markers. Does this patch make that easier to do? Without writing custom 
functions?

> I then want the C-x p family of commands
> to allow a choice of inner project or any of the associated
> super-projects.

Please avoid mixing feature requests. I already said that "choice of 
inner or outer" is out of scope for this, but it's easily implemented on 
top.

> I can't understand from the patch alone if it solves this case, but it
> seems that it doesn't come near.  If not for any other reason, I can't
> always use marker files.

Did you prefer the other patch in this report?

Or do you perhaps see an easy way to augment this patch to cover the 
remaining cases? Perhaps of project-vc-extra-root-markers also accepted 
absolute directory names, or if an additional variable did that.

> I do see that in your patch more and more things appeari under the
> existing VC type, which I think is growing too much.  The comment itself
> admits that "VC" becomes "VC and etc." -- this is not a good sign.
> 
> The patch seems to be trying very hard not to create a new project type.

That's called "tradeoffs". Previous patches, if you saw them, made 
different tradeoffs with different downsides.

> But if a new subproject type was created with a link to a previously
> found super-project.  One could easily e.g. reuse the super-project's
> ignore rules etc in the sub-project.

For us to be able to discuss the alternatives, you'd have to read the 
previous comments on this report. Or I guess you can post a reasonably 
complete and functional alternative patch and we'd discuss the tradeoffs 
there.

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

On Sat, Nov 26, 2022, 12:30 Dmitry Gutov <dgutov@yandex.ru> wrote:

>
> > My use case is the following: I'm interested in being able to designate
> > projects (through various means, not only marker files) that may only
> > exist inside other projects.
>
> You previously described your super-project and how you handled it using
> project-find-functions hook with a new element that looked for file
> markers. Does this patch make that easier to do? Without writing custom
> functions?
>

The example i gave did _not_ use file markers. Personally, I can't use
them. I need some elisp way.

> I then want the C-x p family of commands
> > to allow a choice of inner project or any of the associated
> > super-projects.
>
> Please avoid mixing feature requests. I already said that "choice of
> inner or outer" is out of scope for this, but it's easily implemented on
> top.
>

What good are sub and super projects without a way to take advantage of
them? If anything we should focus on the operations first.

I have not seen your other patch. I take it it must have had some drawback
since you superseded it with something else. But post the link, this thread
is too long. I'll look at it on Monday if I have time.

[-- Attachment #2: Type: text/html, Size: 1908 bytes --]

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2413 bytes --]

On 26/11/22 21:23, João Távora wrote:
> On Sat, Nov 26, 2022, 12:30 Dmitry Gutov <dgutov@yandex.ru 
> <mailto:dgutov@yandex.ru>> wrote:
> 
> 
>      > My use case is the following: I'm interested in being able to
>     designate
>      > projects (through various means, not only marker files) that may only
>      > exist inside other projects.
> 
>     You previously described your super-project and how you handled it
>     using
>     project-find-functions hook with a new element that looked for file
>     markers. Does this patch make that easier to do? Without writing custom
>     functions?
> 
> 
> The example i gave did _not_ use file markers. Personally, I can't use 
> them. I need some elisp way.

Please elaborate. Does it mean that those subprojects are chosen 
manually and don't have "packages.jon" or etc exactly (or that too many 
subprojects in that same project would, undesirably, contain the same 
files)?

Would being able to set to absolute file names (directories) help? Or is 
that too awkward? Worst case, we could also add the new option from 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#85, the result see 
attached.

>      > I then want the C-x p family of commands
>      > to allow a choice of inner project or any of the associated
>      > super-projects.
> 
>     Please avoid mixing feature requests. I already said that "choice of
>     inner or outer" is out of scope for this, but it's easily
>     implemented on
>     top.
> 
> 
> What good are sub and super projects without a way to take advantage of 
> them? If anything we should focus on the operations first.

As I have demonstrated, the features are orthogonal.

Let's do it piece by piece, otherwise this bug might stay open another 
couple of years.

> I have not seen your other patch. I take it it must have had some 
> drawback since you superseded it with something else. But post the link, 
> this thread is too long. I'll look at it on Monday if I have time.

That would be https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#85

The downsides we have already discussed: having to customize every 
"super" project individually is a pain, people more often seem to prefer 
to use "markers", the set of which is customized once.

So we have to support "markers" anyway, hence it makes sense to try to 
make do with them only. But here's how it would look if we try to 
support both approaches.

[-- Attachment #2: project-vc-extra-root-markers-and-subprojects.diff --]
[-- Type: text/x-patch, Size: 6594 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 5b8648031fb..05ba631e52f 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -66,6 +66,9 @@
 ;; files, but supports additions to the list using the user option
 ;; `project-vc-ignores' (usually through .dir-locals.el).
 ;;
+;; At this point the name might as well be an abbreviation for "VC and
+;; Etc", see the variable `project-vc-extra-root-markers'.
+;;
 ;; Utils:
 ;;
 ;; `project-combine-directories' and `project-subtract-directories',
@@ -411,6 +414,51 @@ project-vc-name
   :version "29.1"
   :safe #'stringp)
 
+;; Not using regexps because these wouldn't work in Git pathspecs, in
+;; case we decide we need to be able to list subprojects.
+(defcustom project-vc-extra-root-markers nil
+  "List of additional markers to signal project roots.
+
+A marker is either a base file name or a glob pattern for such.
+
+A directory containing such a marker file or a file matching a
+marker pattern will be recognized as the root of a VC project.
+
+Example values: \".dir-locals.el\", \"package.json\", \"pom.xml\",
+\"requirements.txt\", \"Gemfile\", \"*.gemspec\", \"autogen.sh\".
+
+These will be used in addition to regular directory markers such
+as \".git\", \".hg\", and so on, depending on the value of
+`vc-handled-backends'.  It is most useful when a VC project has
+subdirectories inside it that need to be considered as separate
+projects.  It can also be used for projects outside of VC
+repositories.
+
+In either case, their behavior will still obey the relevant
+variables, such as `project-vc-ignores' or `project-vc-name'."
+  :type 'list
+  :version "29.1"
+  :safe (lambda (val) (and (listp val) (cl-every #'stringp val))))
+
+(defcustom project-vc-subprojects nil
+  "List of relative directory names to consider separate projects.
+Each entry should a string, name of a subproject root directory
+relative to the VC project root.
+
+Whenever a VC project root detected according to the usual
+conditions contains a subdirectory from that list, that
+subdirectory will be recognized as the root of a separate VC
+project as well.
+
+One would usually set this variable through the dir-locals
+mechanism.
+
+If subprojects are Git submodules, you can use the variable
+`project-vc-merge-submodules' instead."
+  :type 'list
+  :version "29.1"
+  :safe (lambda (val) (and (listp val) (seq-every-p #'stringp val))))
+
 ;; FIXME: Using the current approach, major modes are supposed to set
 ;; this variable to a buffer-local value.  So we don't have access to
 ;; the "external roots" of language A from buffers of language B, which
@@ -447,29 +495,59 @@ project-vc-external-roots-function
 backend implementation of `project-external-roots'.")
 
 (defun project-try-vc (dir)
+  (defvar vc-svn-admin-directory)
+  (require 'vc-svn)
+  ;; FIXME: Learn to invalidate when the value of
+  ;; `project-vc-merge-submodules' or `project-vc-extra-root-markers'
+  ;; changes.
   (or (vc-file-getprop dir 'project-vc)
-      (let* ((backend (ignore-errors (vc-responsible-backend dir)))
+      (let* ((backend-markers-alist `((Git . ".git")
+                                      (Hg . ".hg")
+                                      (Bzr . ".bzr")
+                                      (SVN . ,vc-svn-admin-directory)
+                                      (DARCS . "_darcs")
+                                      (Fossil . ".fslckout")))
+             (backend-markers
+              (delete
+               nil
+               (mapcar
+                (lambda (b) (assoc-default b backend-markers-alist))
+                vc-handled-backends)))
+             (marker-re
+              (mapconcat
+               (lambda (m) (format "\\(%s\\)" (wildcard-to-regexp m)))
+               (append backend-markers project-vc-extra-root-markers)
+               "\\|"))
+             (locate-dominating-stop-dir-regexp
+              (or vc-ignore-dir-regexp locate-dominating-stop-dir-regexp))
+             last-matches
              (root
-              (pcase backend
-                ('Git
-                 ;; Don't stop at submodule boundary.
-                 (or (vc-file-getprop dir 'project-git-root)
-                     (let ((root (vc-call-backend backend 'root dir)))
-                       (vc-file-setprop
-                        dir 'project-git-root
-                        (if (and
-                             ;; FIXME: Invalidate the cache when the value
-                             ;; of this variable changes.
-                             project-vc-merge-submodules
-                             (project--submodule-p root))
-                            (let* ((parent (file-name-directory
-                                            (directory-file-name root))))
-                              (vc-call-backend backend 'root parent))
-                          root)))))
-                ('nil nil)
-                (_ (ignore-errors (vc-call-backend backend 'root dir)))))
+              (locate-dominating-file
+               dir
+               (lambda (d)
+                 (setq last-matches (directory-files d nil marker-re t 100)))))
+             (backend
+              (cl-find-if
+               (lambda (b)
+                 (member (assoc-default b backend-markers-alist)
+                         last-matches))
+               vc-handled-backends))
              project)
+        (when (and
+               (eq backend 'Git)
+               project-vc-merge-submodules
+               (project--submodule-p root))
+          (let* ((parent (file-name-directory (directory-file-name root))))
+            (setq root (vc-call-backend 'Git 'root parent))))
         (when root
+          (let* ((relative-dir (file-relative-name dir root))
+                 (subproject (seq-find
+                              (lambda (sub-dir)
+                                (string-prefix-p (file-name-as-directory sub-dir)
+                                                 relative-dir))
+                              project-vc-subprojects)))
+            (and subproject
+                 (setq root (concat root subproject))))
           (setq project (list 'vc backend root))
           ;; FIXME: Cache for a shorter time.
           (vc-file-setprop dir 'project-vc project)
@@ -626,7 +704,8 @@ project-ignores
   (let* ((root (nth 2 project))
          backend)
     (append
-     (when (file-equal-p dir root)
+     (when (and backend
+                (file-equal-p dir root))
        (setq backend (cadr project))
        (delq
         nil

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 26/11/22 21:23, João Távora wrote:
>> On Sat, Nov 26, 2022, 12:30 Dmitry Gutov <dgutov@yandex.ru
>> <mailto:dgutov@yandex.ru>> wrote:
>>      > My use case is the following: I'm interested in being able to
>>     designate
>>      > projects (through various means, not only marker files) that may only
>>      > exist inside other projects.
>>     You previously described your super-project and how you handled
>> it
>>     using
>>     project-find-functions hook with a new element that looked for file
>>     markers. Does this patch make that easier to do? Without writing custom
>>     functions?
>> The example i gave did _not_ use file markers. Personally, I can't
>> use them. I need some elisp way.
>
> Please elaborate.

I've already elaborated, with actual code.

https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01505.html
https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01530.html

> Does it mean that those subprojects are chosen manually and don't have
> "packages.jon" or etc exactly (or that too many subprojects in that
> same project would, undesirably, contain the same files)?

OK, one last time: packages.json and i.e. monorepos that have a
developing collection of similarly structured NPM packages that move
around is good case for marker files, undoubtedly.  I'm not putting that
into question.

But many times that's not the case and you have a big project in which a
mostly fixed heterogenous collection of sub-hierarchies exist and you
would like to designate as those as subprojects.  In the latter case,
marker files are useless, uselessly slow and perhaps even impossible.

In _both_ cases, it's very useful to have project operations let the
user choose the target: super-project or sub-project (or "parent
project", "outer project", "nested project", "inner project": I don't
care too much about the nomenclature).

> Would being able to set to absolute file names (directories) help? Or
> would that be too awkward?

That's more or less the idea, but they don't need to be absolute file
names which is indeed awkward See project-sub-project-prefixes in the
code I posted.  project-sub-project-prefixes can even be a regex pattern
applied on the super-project's root.  This describes the heterogenous
collection economically and robustly.  It is then typically set
dir-locally, with either a .dir-locals.el file or with
dir-locals-set-class-variables.

Of course, the member of project-find-functions that consults
project-sub-project-prefixes needs to be aware of containing
super-projects found by some other means (marker files included).  See
the code I posted to emacs-devel for a possible implementation.

João

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 29/11/2022 11:46, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 26/11/22 21:23, João Távora wrote:
>>> On Sat, Nov 26, 2022, 12:30 Dmitry Gutov <dgutov@yandex.ru
>>> <mailto:dgutov@yandex.ru>> wrote:
>>>       > My use case is the following: I'm interested in being able to
>>>      designate
>>>       > projects (through various means, not only marker files) that may only
>>>       > exist inside other projects.
>>>      You previously described your super-project and how you handled
>>> it
>>>      using
>>>      project-find-functions hook with a new element that looked for file
>>>      markers. Does this patch make that easier to do? Without writing custom
>>>      functions?
>>> The example i gave did _not_ use file markers. Personally, I can't
>>> use them. I need some elisp way.
>>
>> Please elaborate.
> 
> I've already elaborated, with actual code.
> 
> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01505.html
> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01530.html

These answered the question of *what* you want to do, not *why*.

>> Does it mean that those subprojects are chosen manually and don't have
>> "packages.jon" or etc exactly (or that too many subprojects in that
>> same project would, undesirably, contain the same files)?
> 
> OK, one last time: packages.json and i.e. monorepos that have a
> developing collection of similarly structured NPM packages that move
> around is good case for marker files, undoubtedly.  I'm not putting that
> into question.
> 
> But many times that's not the case and you have a big project in which a
> mostly fixed heterogenous collection of sub-hierarchies exist and you
> would like to designate as those as subprojects.  In the latter case,
> marker files are useless, uselessly slow and perhaps even impossible.

I understand that in theory, it's just it's often possible to solve the 
problem with the tools at hand (see my latest reply on emacs-devel about 
the Emacs doc/ subdirectory). So I figured to ask about your particulars.

> In _both_ cases, it's very useful to have project operations let the
> user choose the target: super-project or sub-project (or "parent
> project", "outer project", "nested project", "inner project": I don't
> care too much about the nomenclature).

Yes. But separate feature.

>> Would being able to set to absolute file names (directories) help? Or
>> would that be too awkward?
> 
> That's more or less the idea, but they don't need to be absolute file
> names which is indeed awkward See project-sub-project-prefixes in the
> code I posted.  project-sub-project-prefixes can even be a regex pattern
> applied on the super-project's root.  This describes the heterogenous
> collection economically and robustly.  It is then typically set
> dir-locally, with either a .dir-locals.el file or with
> dir-locals-set-class-variables.

I asked about absolute file names because those would be easier 
(semantically) to cram into the same user option as the markers.

Otherwise we'd need a separate option for those.

Although... if we insisted on using the format like "./abc/def/", that 
would also put those values into a different category. The handling 
logic would need to be different just the same.

> Of course, the member of project-find-functions that consults
> project-sub-project-prefixes needs to be aware of containing
> super-projects found by some other means (marker files included).  See
> the code I posted to emacs-devel for a possible implementation.

Have you tried the patch that I sent in the GP email?

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>>> Please elaborate.
>> I've already elaborated, with actual code.
>> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01505.html
>> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01530.html
>
> These answered the question of *what* you want to do, not *why*.

I've described that numerous times as well.  I want to be able to grep,
compile, etc in the either the super or the sub project.  I've described
this to you so many times it's astonishing you keep asking me to
elaborate.

> I understand that in theory,

Thank heavens for that!

> it's just it's often possible to solve the problem with the tools at
> hand (see my latest reply on emacs-devel about the Emacs doc/
> subdirectory).

I've seen it, and it's not a good solution.  Will reply there.

João

bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov]

On 30/11/2022 00:06, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>>>> Please elaborate.
>>> I've already elaborated, with actual code.
>>> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01505.html
>>> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01530.html
>>
>> These answered the question of *what* you want to do, not *why*.
> 
> I've described that numerous times as well.  I want to be able to grep,
> compile, etc in the either the super or the sub project.  I've described
> this to you so many times it's astonishing you keep asking me to
> elaborate.

Should I repeat, for the 1000th time, that it will be possible to add no 
matter which of the "subproject" approaches we take?

I've even posted the patch.

>> I understand that in theory,
> 
> Thank heavens for that!
> 
>> it's just it's often possible to solve the problem with the tools at
>> hand (see my latest reply on emacs-devel about the Emacs doc/
>> subdirectory).
> 
> I've seen it, and it's not a good solution.  Will reply there.

Okay.

Should I wait for you to finally try the patch from the GGP email? Or is 
that too much to ask?

 From what I see of your responses, you didn't read it either.