bug#72300: project.el: detect newly created project contained within another

bug#72300: project.el: detect newly created project contained within another

[Federico Tedin via Bug reports for GNU Emacs, the Swiss army knife of text editors]

In Emacs master e56e4b345a2, `emacs -q`:

I'm having problems trying to make project.el detect a new project that
is contained in the directory of another project.

I have a directory called 'scratch' which contains a '.git' directory, a
file 'test.py' and a directory 'foo'. The 'foo' directory contains a
file called 'foo.py'.


~/scratch/
    .git/
    main.py
    foo/
        foo.py


If I open 'main.py', `(project-current)' evals to the expected: `(vc Git
"~/scratch")'.

If I open 'foo.py', `(project-current)' also evals to `(vc Git
"~/scratch")', which is expected.

However if now I cd into 'foo/' and run `git init`, then I would expect
project.el to now consider 'foo.py' to be in another project - `(vc Git
"~/scratch/foo")'. However, if I evaluate `(project-current)' when
visiting 'foo.py', I still get `(vc Git "~/scratch")'.

If I kill the buffer visiting 'foo.py' and open the file again, I get
the same result.

Interestingly, if I run 'M-x project-remember-projects-under' with
'~/scratch/foo' as path, it does inform me that the new project has been
found. However visiting 'foo.py` still results in `(vc Git "~/scratch")'
as the current project.

If I restart Emacs then the problem is solved; 'foo.py' is correctly
filed under project `(vc Git "~/scratch/foo")'.

The fact that this works correctly after restarting makes me think
that there must be some runtime state set up that is preventing the
desired behaviour to happen.

bug#72300: project.el: detect newly created project contained within another

[Eli Zaretskii]

> Date: Thu, 25 Jul 2024 21:54:24 +0200
> From:  Federico Tedin via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> In Emacs master e56e4b345a2, `emacs -q`:
> 
> I'm having problems trying to make project.el detect a new project that
> is contained in the directory of another project.
> 
> I have a directory called 'scratch' which contains a '.git' directory, a
> file 'test.py' and a directory 'foo'. The 'foo' directory contains a
> file called 'foo.py'.
> 
> 
> ~/scratch/
>     .git/
>     main.py
>     foo/
>         foo.py
> 
> 
> If I open 'main.py', `(project-current)' evals to the expected: `(vc Git
> "~/scratch")'.
> 
> If I open 'foo.py', `(project-current)' also evals to `(vc Git
> "~/scratch")', which is expected.
> 
> However if now I cd into 'foo/' and run `git init`, then I would expect
> project.el to now consider 'foo.py' to be in another project - `(vc Git
> "~/scratch/foo")'. However, if I evaluate `(project-current)' when
> visiting 'foo.py', I still get `(vc Git "~/scratch")'.
> 
> If I kill the buffer visiting 'foo.py' and open the file again, I get
> the same result.
> 
> Interestingly, if I run 'M-x project-remember-projects-under' with
> '~/scratch/foo' as path, it does inform me that the new project has been
> found. However visiting 'foo.py` still results in `(vc Git "~/scratch")'
> as the current project.
> 
> If I restart Emacs then the problem is solved; 'foo.py' is correctly
> filed under project `(vc Git "~/scratch/foo")'.
> 
> The fact that this works correctly after restarting makes me think
> that there must be some runtime state set up that is preventing the
> desired behaviour to happen.

Dmitry, any comments or suggestions?

bug#72300: project.el: detect newly created project contained within another

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 2481 bytes --]

This appears to be intentional behavior of project's caching implemented
via (vc-file-getprop dir 'project-vc) and
(vc-file-setprop dir 'project-vc project) in project-try-vc. There is no
facility, public API or private, to clear the cache en-masse. One could
reset the cache via clearing the vector vc-file-prop-obarray
(setq vc-file-prop-obarray (make-vector 17 0)) in the absence of an API.
You can observe what's in your vc-file-prop-obarray for yourself before
taking this action.

Hope that helps,

-Stephane

On Sun, Aug 4, 2024 at 4:16 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Thu, 25 Jul 2024 21:54:24 +0200
> > From:  Federico Tedin via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> >
> > In Emacs master e56e4b345a2, `emacs -q`:
> >
> > I'm having problems trying to make project.el detect a new project that
> > is contained in the directory of another project.
> >
> > I have a directory called 'scratch' which contains a '.git' directory, a
> > file 'test.py' and a directory 'foo'. The 'foo' directory contains a
> > file called 'foo.py'.
> >
> >
> > ~/scratch/
> >     .git/
> >     main.py
> >     foo/
> >         foo.py
> >
> >
> > If I open 'main.py', `(project-current)' evals to the expected: `(vc Git
> > "~/scratch")'.
> >
> > If I open 'foo.py', `(project-current)' also evals to `(vc Git
> > "~/scratch")', which is expected.
> >
> > However if now I cd into 'foo/' and run `git init`, then I would expect
> > project.el to now consider 'foo.py' to be in another project - `(vc Git
> > "~/scratch/foo")'. However, if I evaluate `(project-current)' when
> > visiting 'foo.py', I still get `(vc Git "~/scratch")'.
> >
> > If I kill the buffer visiting 'foo.py' and open the file again, I get
> > the same result.
> >
> > Interestingly, if I run 'M-x project-remember-projects-under' with
> > '~/scratch/foo' as path, it does inform me that the new project has been
> > found. However visiting 'foo.py` still results in `(vc Git "~/scratch")'
> > as the current project.
> >
> > If I restart Emacs then the problem is solved; 'foo.py' is correctly
> > filed under project `(vc Git "~/scratch/foo")'.
> >
> > The fact that this works correctly after restarting makes me think
> > that there must be some runtime state set up that is preventing the
> > desired behaviour to happen.
>
> Dmitry, any comments or suggestions?
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3827 bytes --]

bug#72300: project.el: detect newly created project contained within another

[Federico Tedin via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ship Mints <shipmints@gmail.com> writes:

> This appears to be intentional behavior of project's caching implemented via (vc-file-getprop dir 'project-vc) and
> (vc-file-setprop dir 'project-vc project) in project-try-vc. There is no facility, public API or private, to clear the cache en-masse. One could reset the
> cache via clearing the vector vc-file-prop-obarray (setq vc-file-prop-obarray (make-vector 17 0)) in the absence of an API. You can observe what's
> in your vc-file-prop-obarray for yourself before taking this action.

Ah wow, indeed setting that variable to (make-vector 17 0) does solve
the problem. Thanks! Now I wonder if this could/should be exposed to the
user somehow.

bug#72300: project.el: detect newly created project contained within another

[Dmitry Gutov]

Hi!

On 05/08/2024 20:18, Ship Mints wrote:
> (vc-file-setprop dir 'project-vc project) in project-try-vc. There is no 
> facility, public API or private, to clear the cache en-masse. One could 
> reset the cache via clearing the vector vc-file-prop-obarray 
> (setq vc-file-prop-obarray (make-vector 17 0)) in the absence of an API. 
> You can observe what's in your vc-file-prop-obarray for yourself before 
> taking this action.

That's right. One step toward that goal would be moving the cache to 
some other data structure - possibly a tree-like one, to also be able to 
short-circuit the upward directory searches.

Cache invalidation is a sore point, though: the directory tree can 
change behind the scenes outside Emacs, so unless the caching is 
disabled the other complete solutions would rely on something like 
filenotify.

OT2H if we're okay with supporting only manual clears e.g. using 'M-x 
project-forget-project' or 'M-x project-forget-projects-under', that 
could be implemented easily enough. The current vc-file-prop-obarray 
structure could be refreshed with a full scan.

bug#72300: project.el: detect newly created project contained within another

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 2654 bytes --]

A good step is awareness for users and invalidating the cache via
project-forget methods is a good idea. I'd also offer a direct function to
invoke to invalidate the cache for programmatic use.

vc caching, longer term, may need to consider a few more complex use cases
such as git repos with both submodules that are considered extensions of
the base project and submodules which are not. A concrete example I see
often is a "mono repo" structure with core server and library code but with
web and mobile front end code in submodules that are treated as part of the
project proper BUT with submodules for vendor/third-party code that are
not. A question here would be which parts of the tree belong to which
cached vc root.

Another use case I see is working on many unrelated projects/repos across a
variety of clients all in the same Emacs session and with perhaps 100+
buffers/files open (as I pretty much have right now), a 17-element cache
won't be sufficient? Should the cache be for parent directories and not for
file names? With files, it gets full fast. Mine is full right now with
files most of which share the same repo root and some that don't. I have
wondered whether an implementation would be better as directory variables?
Cache invalidation without timestamps on .dir-locals.el files remain the
same but directory variable treatment might be more natural to Emacs users?

-Stephane

On Mon, Aug 12, 2024 at 9:43 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> Hi!
>
> On 05/08/2024 20:18, Ship Mints wrote:
> > (vc-file-setprop dir 'project-vc project) in project-try-vc. There is no
> > facility, public API or private, to clear the cache en-masse. One could
> > reset the cache via clearing the vector vc-file-prop-obarray
> > (setq vc-file-prop-obarray (make-vector 17 0)) in the absence of an API.
> > You can observe what's in your vc-file-prop-obarray for yourself before
> > taking this action.
>
> That's right. One step toward that goal would be moving the cache to
> some other data structure - possibly a tree-like one, to also be able to
> short-circuit the upward directory searches.
>
> Cache invalidation is a sore point, though: the directory tree can
> change behind the scenes outside Emacs, so unless the caching is
> disabled the other complete solutions would rely on something like
> filenotify.
>
> OT2H if we're okay with supporting only manual clears e.g. using 'M-x
> project-forget-project' or 'M-x project-forget-projects-under', that
> could be implemented easily enough. The current vc-file-prop-obarray
> structure could be refreshed with a full scan.
>

[-- Attachment #2: Type: text/html, Size: 3487 bytes --]

bug#72300: project.el: detect newly created project contained within another

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 3625 bytes --]

As some food for thought, we've used git attributes via "git check-attr" to
record custom metadata at the repository level to provide hints to tooling.
As project.el and other tools evolve, perhaps we can consider the cases
where users can supply functions that influence behavior where one use case
is based on things like git attributes. The case in hand is whether a
submodule should be considered external to a project.el project such as the
vendor/third-party submodule I'd mentioned. We use the
default project-vc-merge-submodules t and our exceptions could be encoded
via a custom git attribute and indicated via a predicate function, perhaps.
In the absence of formal guidelines, and git does not seem to define
"reserved words" for naming, we prefix custom attributes with an underscore.

On Tue, Aug 13, 2024 at 9:31 AM Ship Mints <shipmints@gmail.com> wrote:

> A good step is awareness for users and invalidating the cache via
> project-forget methods is a good idea. I'd also offer a direct function to
> invoke to invalidate the cache for programmatic use.
>
> vc caching, longer term, may need to consider a few more complex use cases
> such as git repos with both submodules that are considered extensions of
> the base project and submodules which are not. A concrete example I see
> often is a "mono repo" structure with core server and library code but with
> web and mobile front end code in submodules that are treated as part of the
> project proper BUT with submodules for vendor/third-party code that are
> not. A question here would be which parts of the tree belong to which
> cached vc root.
>
> Another use case I see is working on many unrelated projects/repos across
> a variety of clients all in the same Emacs session and with perhaps 100+
> buffers/files open (as I pretty much have right now), a 17-element cache
> won't be sufficient? Should the cache be for parent directories and not for
> file names? With files, it gets full fast. Mine is full right now with
> files most of which share the same repo root and some that don't. I have
> wondered whether an implementation would be better as directory variables?
> Cache invalidation without timestamps on .dir-locals.el files remain the
> same but directory variable treatment might be more natural to Emacs users?
>
> -Stephane
>
> On Mon, Aug 12, 2024 at 9:43 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
>> Hi!
>>
>> On 05/08/2024 20:18, Ship Mints wrote:
>> > (vc-file-setprop dir 'project-vc project) in project-try-vc. There is
>> no
>> > facility, public API or private, to clear the cache en-masse. One could
>> > reset the cache via clearing the vector vc-file-prop-obarray
>> > (setq vc-file-prop-obarray (make-vector 17 0)) in the absence of an
>> API.
>> > You can observe what's in your vc-file-prop-obarray for yourself before
>> > taking this action.
>>
>> That's right. One step toward that goal would be moving the cache to
>> some other data structure - possibly a tree-like one, to also be able to
>> short-circuit the upward directory searches.
>>
>> Cache invalidation is a sore point, though: the directory tree can
>> change behind the scenes outside Emacs, so unless the caching is
>> disabled the other complete solutions would rely on something like
>> filenotify.
>>
>> OT2H if we're okay with supporting only manual clears e.g. using 'M-x
>> project-forget-project' or 'M-x project-forget-projects-under', that
>> could be implemented easily enough. The current vc-file-prop-obarray
>> structure could be refreshed with a full scan.
>>
>

[-- Attachment #2: Type: text/html, Size: 4749 bytes --]

bug#72300: project.el: detect newly created project contained within another

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2642 bytes --]

On 13/08/2024 16:31, Ship Mints wrote:
> A good step is awareness for users and invalidating the cache via 
> project-forget methods is a good idea. 

For posterity: making it a method of an object doesn't seem to work 
since the directory might have changed enough that it's not possible to 
recreate the project instance to call it on.

 > I'd also offer a direct function
 > to invoke to invalidate the cache for programmatic use.

project-forget-project is a Lisp function which could be used both 
interactively and from code. Attached is a PoC/WIP patch which adds 
cache invalidation this way. Not entirely happy with the approach yet, 
but I'd like to checkpoint it here.

> vc caching, longer term, may need to consider a few more complex use 
> cases such as git repos with both submodules that are considered 
> extensions of the base project and submodules which are not. A concrete 
> example I see often is a "mono repo" structure with core server and 
> library code but with web and mobile front end code in submodules that 
> are treated as part of the project proper BUT with submodules for 
> vendor/third-party code that are not. A question here would be which 
> parts of the tree belong to which cached vc root.

Very tangentially, we've just added support for speeding up file listing 
for projects using "sparse index". That one's for a different structure 
and repos above certain size, though.

> Another use case I see is working on many unrelated projects/repos 
> across a variety of clients all in the same Emacs session and with 
> perhaps 100+ buffers/files open (as I pretty much have right now), a 17- 
> element cache won't be sufficient? Should the cache be for parent 
> directories and not for file names? With files, it gets full fast. Mine 
> is full right now with files most of which share the same repo root and 
> some that don't.

These are all internal details, but:

* 17 is the default size, then it grows,
* project-try-vc caches only directories, using its own key,
* The rest of the data you're seeing is for files and their VC statuses, 
that's separate info.

But indeed it's a bit too much intermixing, project-vc's cache should 
move to its own structure first. Stay tuned.

> I have wondered whether an implementation would be 
> better as directory variables? Cache invalidation without timestamps 
> on .dir-locals.el files remain the same but directory variable treatment 
> might be more natural to Emacs users?

Directory variables are a thing, but not in the same sense that 
file-local variables are, in particular there is no object to save a 
"directory local" to at runtime.

[-- Attachment #2: project-forget-functions.diff --]
[-- Type: text/x-patch, Size: 1827 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 599a350e5ce..9cc70335afc 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -193,6 +193,10 @@ project-find-functions
   'project-current-directory-override
   "29.1")
 
+(defvar project-forget-functions (list #'project-vc-forget-directory)
+  "Special hook to clear cache for a directory tree.
+Each function on this hook is passed a string, the directory file name.")
+
 (defvar project-current-directory-override nil
   "Value to use instead of `default-directory' when detecting the project.
 When it is non-nil, `project-current' will always skip prompting too.")
@@ -599,6 +603,13 @@ project-try-vc
           (vc-file-setprop dir 'project-vc project)
           project))))
 
+(defun project-vc-forget-directory (root)
+  (obarray-map
+   (lambda (sym)
+     (when (string-prefix-p root (symbol-name sym))
+       (vc-file-setprop (symbol-name sym) 'project-vc nil)))
+   vc-file-prop-obarray))
+
 (defun project--submodule-p (root)
   ;; XXX: We only support Git submodules for now.
   ;;
@@ -1875,6 +1886,7 @@ project-forget-project
 PROJECT-ROOT is the root directory of a known project listed in
 the project list."
   (interactive (list (funcall project-prompter)))
+  (run-hook-with-args 'project-forget-functions project-root)
   (project--remove-from-project-list
    project-root "Project `%s' removed from known projects"))
 
@@ -1999,6 +2011,8 @@ project-remember-projects-under
 projects."
   (interactive "DDirectory: \nP")
   (project--ensure-read-project-list)
+  ;; Clear the cache for DIR and below, in case of prior visit.
+  (run-hook-with-args 'project-forget-functions dir)
   (let ((dirs (if recursive
                   (directory-files-recursively dir "" t)
                 (directory-files dir t)))

bug#72300: project.el: detect newly created project contained within another

[Dmitry Gutov]

On 13/08/2024 17:50, Ship Mints wrote:
> As some food for thought, we've used git attributes via "git check-attr" 
> to record custom metadata at the repository level to provide hints to 
> tooling. As project.el and other tools evolve, perhaps we can consider 
> the cases where users can supply functions that influence behavior where 
> one use case is based on things like git attributes. The case in hand is 
> whether a submodule should be considered external to a project.el 
> project such as the vendor/third-party submodule I'd mentioned. We use 
> the default project-vc-merge-submodules t and our exceptions could be 
> encoded via a custom git attribute and indicated via a predicate 
> function, perhaps. In the absence of formal guidelines, and git does not 
> seem to define "reserved words" for naming, we prefix custom attributes 
> with an underscore.

Git attributes is a possible approach, with a downside of extra process 
calls, which over Tramp (for example) would mean additional latency.

How about we just support filtering out submodules using the 
project-vc-ignores var? Which can be assigned in .dir-locals.el or 
through other means.

I kind of hoped this would work automatically, it does not, but adding 
the extra comparisons against the list of ignores seems natural enough.

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 599a350e5ce..79fcfb65f87 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -720,7 +731,9 @@ project--vc-list-files
           (let ((sub-files
                  (mapcar
                   (lambda (module)
-                   (when (file-directory-p module)
+                   (when (and (file-directory-p module)
+                              ;; TODO: Support real globs.
+                              (not (member module extra-ignores)))
                       (let ((sub-files
                              (project--vc-list-files
                               (concat default-directory module)

bug#72300: project.el: detect newly created project contained within another

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 2571 bytes --]

On Sun, Sep 29, 2024 at 9:50 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> On 13/08/2024 17:50, Ship Mints wrote:
> > As some food for thought, we've used git attributes via "git check-attr"
> > to record custom metadata at the repository level to provide hints to
> > tooling. As project.el and other tools evolve, perhaps we can consider
> > the cases where users can supply functions that influence behavior where
> > one use case is based on things like git attributes. The case in hand is
> > whether a submodule should be considered external to a project.el
> > project such as the vendor/third-party submodule I'd mentioned. We use
> > the default project-vc-merge-submodules t and our exceptions could be
> > encoded via a custom git attribute and indicated via a predicate
> > function, perhaps. In the absence of formal guidelines, and git does not
> > seem to define "reserved words" for naming, we prefix custom attributes
> > with an underscore.
>
> Git attributes is a possible approach, with a downside of extra process
> calls, which over Tramp (for example) would mean additional latency.
>

(defun project--value-in-dir (var dir) already incurs latency over tramp,
right?


> How about we just support filtering out submodules using the
> project-vc-ignores var? Which can be assigned in .dir-locals.el or
> through other means.
>

Seems simple. Perhaps an abnormal hook so people can customize buffer local
hook via dir locals? If they want to use git attributes, they could
integrate that approach as an added/replacement function. The function list
could default to a function that implements current behavior.


> I kind of hoped this would work automatically, it does not, but adding
> the extra comparisons against the list of ignores seems natural enough.
>
> diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
> index 599a350e5ce..79fcfb65f87 100644
> --- a/lisp/progmodes/project.el
> +++ b/lisp/progmodes/project.el
> @@ -720,7 +731,9 @@ project--vc-list-files
>            (let ((sub-files
>                   (mapcar
>                    (lambda (module)
> -                   (when (file-directory-p module)
> +                   (when (and (file-directory-p module)
> +                              ;; TODO: Support real globs.
> +                              (not (member module extra-ignores)))
>                        (let ((sub-files
>                               (project--vc-list-files
>                                (concat default-directory module)
>
>

[-- Attachment #2: Type: text/html, Size: 3676 bytes --]

bug#72300: project.el: detect newly created project contained within another

[Dmitry Gutov]

On 30/09/2024 17:31, Ship Mints wrote:

>     Git attributes is a possible approach, with a downside of extra process
>     calls, which over Tramp (for example) would mean additional latency.
> 
> 
> (defun project--value-in-dir (var dir)already incurs latency over tramp, 
> right?

Yep, but almost every other separate I/O adds to it. So with other 
things equal, I'd prefer solutions with fewer disk reads - at least for 
the default behavior.

>     How about we just support filtering out submodules using the
>     project-vc-ignores var? Which can be assigned in .dir-locals.el or
>     through other means.
> 
> 
> Seems simple. Perhaps an abnormal hook so people can customize buffer 
> local hook via dir locals? If they want to use git attributes, they 
> could integrate that approach as an added/replacement function. The 
> function list could default to a function that implements current behavior.

I'd like to say yes, but what would be a good place and name for that hook?

Note there is an existing hook called before-hack-local-variables-hook 
which allows one to alter the applied local variables. But that one 
might run more often than ideal - perhaps try it out and report back.

Another approach might use a custom project backend which wraps the 
existing project-vc - it would need to provide a custom implementation 
for 'project-ignores' and could delegate the rest of the methods to the 
parent.

BTW, are you asking about using git attributes because there is an 
existing workflow that somehow hooks into it, at some of your clients? 
If not, then editing dir-locals.el to set the value of 
project-vc-ignores directly seems like an easier approach.

bug#72300: project.el: detect newly created project contained within another

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 4375 bytes --]

On Mon, Sep 30, 2024 at 7:10 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> On 30/09/2024 17:31, Ship Mints wrote:
>
> >     Git attributes is a possible approach, with a downside of extra
> process
> >     calls, which over Tramp (for example) would mean additional latency.
> >
> >
> > (defun project--value-in-dir (var dir)already incurs latency over tramp,
> > right?
>
> Yep, but almost every other separate I/O adds to it. So with other
> things equal, I'd prefer solutions with fewer disk reads - at least for
> the default behavior.
>

Indeed. It would likely have to be a shell-command-to-string via a tramp
file name form.


> >     How about we just support filtering out submodules using the
> >     project-vc-ignores var? Which can be assigned in .dir-locals.el or
> >     through other means.
> >
> >
> > Seems simple. Perhaps an abnormal hook so people can customize buffer
> > local hook via dir locals? If they want to use git attributes, they
> > could integrate that approach as an added/replacement function. The
> > function list could default to a function that implements current
> behavior.
>
> I'd like to say yes, but what would be a good place and name for that hook?
>
> Note there is an existing hook called before-hack-local-variables-hook
> which allows one to alter the applied local variables. But that one
> might run more often than ideal - perhaps try it out and report back.
>

I'll dig back into project.el code and take a look when I have some focus
to think this through.


> Another approach might use a custom project backend which wraps the
> existing project-vc - it would need to provide a custom implementation
> for 'project-ignores' and could delegate the rest of the methods to the
> parent.
>

Also possible. I wonder if this is what Spencer wound up doing to defray
the cost of running "find" on a large hierarchy.


> BTW, are you asking about using git attributes because there is an
> existing workflow that somehow hooks into it, at some of your clients?
> If not, then editing dir-locals.el to set the value of
> project-vc-ignores directly seems like an easier approach.
>

I've used git attributes as semaphores for subdirectories in a monorepo to
identify subprojects (which are not necessarily git submodules, but we have
those, too). This helps with certain tooling external to Emacs. Sadly
git-attr, it doesn't have a simple exit with 0 for success, non-zero for
failure/missing attribute so needs a little parser for its results. The use
cases for git attributes in a monorepo help segregate workflows among front
end, back end, firmware subprojects, among others, where tooling is pretty
different (and developer skill levels) and there are different workflows. I
think I mentioned once before that there is no naming convention for custom
git attributes that I'm aware of so I simply prefix ours with an
underscore. So far, they're just semaphores where we look for check-attr
output of "unspecified" vs. any value for a file/directory of interest.

I haven't been that focused on this aspect in a while. I think there are
some potentially tricky things to think through as I sense that project.el
is becoming more closely tied to vc integration than mere project file
association. In the past (I think on github), I talked about using a meta
project to graft together disparate directory hierarchies for user
convenience beyond just symlinks, though those work, when maintained. e.g.,
how does one find files that are conceptually common to a project that
don't share a directory hierarchy root. The biggest example here is that
user/admin/management documentation lives in cloud drives, while source
code lives locally (via git, mostly). People work across those boundaries.
There are meta project files in each disparate root, .project.meta files
which all contain the same text to graft them together using tools we have.
Some people put in symlinks (like me, I'm lazy) and that helps but not
everyone shares the precise directory hierarchies across every machine
making symlinks harder to deal with when checked into git (and there are
Windows users). This may be a use case that project.el should never
address, not sure.

Only concrete, defensible and not-too-esoteric use cases matter, of course.

-Stephane

[-- Attachment #2: Type: text/html, Size: 6056 bytes --]

bug#72300: project.el: detect newly created project contained within another

[Dmitry Gutov]

On 01/10/2024 23:20, Ship Mints wrote:
> On Mon, Sep 30, 2024 at 7:10 PM Dmitry Gutov <dmitry@gutov.dev 
> <mailto:dmitry@gutov.dev>> wrote:
> 
>     On 30/09/2024 17:31, Ship Mints wrote:
> 
>      >     Git attributes is a possible approach, with a downside of
>     extra process
>      >     calls, which over Tramp (for example) would mean additional
>     latency.
>      >
>      >
>      > (defun project--value-in-dir (var dir)already incurs latency over
>     tramp,
>      > right?
> 
>     Yep, but almost every other separate I/O adds to it. So with other
>     things equal, I'd prefer solutions with fewer disk reads - at least for
>     the default behavior.
> 
> Indeed. It would likely have to be a shell-command-to-string via a tramp 
> file name form.

With a file name handler, then? I suppose it's a possibility.

>      >     How about we just support filtering out submodules using the
>      >     project-vc-ignores var? Which can be assigned in .dir-
>     locals.el or
>      >     through other means.
>      >
>      >
>      > Seems simple. Perhaps an abnormal hook so people can customize
>     buffer
>      > local hook via dir locals? If they want to use git attributes, they
>      > could integrate that approach as an added/replacement function. The
>      > function list could default to a function that implements current
>     behavior.
> 
>     I'd like to say yes, but what would be a good place and name for
>     that hook?
> 
>     Note there is an existing hook called before-hack-local-variables-hook
>     which allows one to alter the applied local variables. But that one
>     might run more often than ideal - perhaps try it out and report back.
> 
> 
> I'll dig back into project.el code and take a look when I have some 
> focus to think this through.

Thanks, looking forward to it.

>     Another approach might use a custom project backend which wraps the
>     existing project-vc - it would need to provide a custom implementation
>     for 'project-ignores' and could delegate the rest of the methods to the
>     parent.
> 
> Also possible. I wonder if this is what Spencer wound up doing to defray 
> the cost of running "find" on a large hierarchy.

Not sure. We're still working on measures aiming to improve performance 
in general, anyway.

>     BTW, are you asking about using git attributes because there is an
>     existing workflow that somehow hooks into it, at some of your clients?
>     If not, then editing dir-locals.el to set the value of
>     project-vc-ignores directly seems like an easier approach.
> 
> 
> I've used git attributes as semaphores for subdirectories in a monorepo 
> to identify subprojects (which are not necessarily git submodules, but 
> we have those, too). This helps with certain tooling external to Emacs. 
> Sadly git-attr, it doesn't have a simple exit with 0 for success, non- 
> zero for failure/missing attribute so needs a little parser for its 
> results. The use cases for git attributes in a monorepo help segregate 
> workflows among front end, back end, firmware subprojects, among others, 
> where tooling is pretty different (and developer skill levels) and there 
> are different workflows. I think I mentioned once before that there is 
> no naming convention for custom git attributes that I'm aware of so I 
> simply prefix ours with an underscore. So far, they're just semaphores 
> where we look for check-attr output of "unspecified" vs. any value for a 
> file/directory of interest.
> 
> I haven't been that focused on this aspect in a while. I think there are 
> some potentially tricky things to think through as I sense that 
> project.el is becoming more closely tied to vc integration than mere 
> project file association.

We can talk about project.el, and we can talk about the project-vc 
backend (the "VC-Aware backend", better called). These two are not the 
same, though naturally related. That's one of the complexities - should 
we add a variable to the one or to the other, for example.

 > In the past (I think on github), I talked > about using a meta 
project to graft together disparate directory
> hierarchies for user convenience beyond just symlinks, though those 
> work, when maintained. e.g., how does one find files that are 
> conceptually common to a project that don't share a directory hierarchy 
> root. The biggest example here is that user/admin/management 
> documentation lives in cloud drives, while source code lives locally 
> (via git, mostly). People work across those boundaries. There are meta 
> project files in each disparate root, .project.meta files which all 
> contain the same text to graft them together using tools we have. Some 
> people put in symlinks (like me, I'm lazy) and that helps but not 
> everyone shares the precise directory hierarchies across every machine 
> making symlinks harder to deal with when checked into git (and there are 
> Windows users). This may be a use case that project.el should never 
> address, not sure.

Maybe not never, but it's currently not very tailored to it. You could 
use project-external-roots, but that part is not so integrated.

Or again, a custom backend could wrap that well enough. Some callers of 
project.el rely on the project being contained in its root dir, but most 
do not.