bug#63870: 29.0.90; project.el can't dynamically populate the project list

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh]

project.el wants a list of known projects, for project-switch-project
and prompting in project-current.

Currently this list is maintained in two ways:
- automatically, by remembering any project the user runs project
  commands within
- manually by the user with project-remember-project and other functions

In both cases, this list is persisted so that it stays around through
Emacs restarts.  All this is good.

But, I often clone new repositories outside of Emacs, and I also have
scripts outside Emacs which make new clones automatically.  It would be
nice for project.el to know about those clones, so I can switch to them
with project-switch-project right away.  Instead, today I usually
manually navigate to those projects the first time, and I'm only able to
use project-switch-project on subsequent times.

These new repos are created in relatively predictable places, so I can
write code which discovers them all.  But I don't have a way to tell
project.el about them.

I could run code on a timer to project-remember-project these projects,
since I create them in predictable locations.  But that would mean
there's a delay between cloning the repo and being able to use it with
project-switch-project, which is annoying especially when I manually
cloned the repo and want to use it immediately.

The new projects are created while Emacs is running, so just remembering
them all at startup doesn't work either.

I'd like a customization point where I can supply a function (or list of
functions) which project-known-project-roots should run to produce an
additional list of project root directories, which should then be
appended to project--list.

I don't need project.el to specifically remember these projects; they'll
be remembered automatically as users use them, and completing-read will
nicely deduplicate the project roots anyway.

Does that seem reasonable?  It would be something like this:

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 04c67710d71..cc05cf460ef 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1679,11 +1679,21 @@ project-prompt-project-name
       (let ((proj (assoc pr-name choices)))
         (if (stringp proj) proj (project-root (cdr proj)))))))
 
+(defcustom project-dynamic-roots '()
+  "List of functions to call to dynamically find projects.
+
+Each is called with no arguments and should return a list of
+project root dirs."
+  :type '(repeat function)
+  :group 'project
+  :version "30.1")
+
 ;;;###autoload
 (defun project-known-project-roots ()
   "Return the list of root directories of all known projects."
   (project--ensure-read-project-list)
-  (mapcar #'car project--list))
+  (flatten-tree (cons (mapcar #'car project--list)
+                      (mapcar #'funcall project-dynamic-roots))))
 
 ;;;###autoload
 (defun project-execute-extended-command ()

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh]

Actually I think a file-notify solution works great.  I have one working
for my use case and I'll post a patch for a generic one soon.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Thu, 15 Jun 2023 15:30:50 -0400
> 
> Actually I think a file-notify solution works great.  I have one working
> for my use case and I'll post a patch for a generic one soon.

Please keep in mind that in our experience watching a repository tree
is problematic (if that is what you had in mind).

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Dmitry Gutov]

Hi!

On 03/06/2023 14:55, Spencer Baugh wrote:
> I'd like a customization point where I can supply a function (or list of
> functions) which project-known-project-roots should run to produce an
> additional list of project root directories, which should then be
> appended to project--list.

Are you sure the existing functions won't cut it? Such as 
project-remember-project and project-forget-project.

The names might seem a little wrong, but keeping in mind that 
project--list is about having a list of projects "remembered" somewhere, 
they're probably fine. And you could let-bind project--list somewhere at 
the top level in your function/command/etc, so the list is not altered 
in the end.

> I don't need project.el to specifically remember these projects; they'll
> be remembered automatically as users use them, and completing-read will
> nicely deduplicate the project roots anyway.

We could add some var like project-list-no-write, so that dynamic 
changes don't get written to disk.

Unless you want to keep the "dynamic" list to be used by regular 
commands, that is.

In that case, the above will probably not suffice. We could go with your 
approach, or even add some project-list-sources hook.

But how would the entries from different sources (e.g. the list file and 
your dynamic list) combined? How will they be sorted?

In case we can't come up with a generic way, we could simply add a 
generic storage abstraction (similar to xref-history-storage), which you 
would override/advise to combine the lists as needed for your usage.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh]

Dmitry Gutov <dmitry@gutov.dev> writes:
> Hi!
>
> On 03/06/2023 14:55, Spencer Baugh wrote:
>> I'd like a customization point where I can supply a function (or list of
>> functions) which project-known-project-roots should run to produce an
>> additional list of project root directories, which should then be
>> appended to project--list.
>
> Are you sure the existing functions won't cut it? Such as
> project-remember-project and project-forget-project.
>
> The names might seem a little wrong, but keeping in mind that
> project--list is about having a list of projects "remembered"
> somewhere, they're probably fine. And you could let-bind project--list
> somewhere at the top level in your function/command/etc, so the list
> is not altered in the end.

Oh, certainly project-{remember,forget}-project work, indeed I've used
them in the patch I just posted.  I think I was unclear about what I
wanted, perhaps that patch clarifies what exactly I was looking for.

>> I don't need project.el to specifically remember these projects; they'll
>> be remembered automatically as users use them, and completing-read will
>> nicely deduplicate the project roots anyway.
>
> We could add some var like project-list-no-write, so that dynamic
> changes don't get written to disk.
>
> Unless you want to keep the "dynamic" list to be used by regular
> commands, that is.
>
> In that case, the above will probably not suffice. We could go with
> your approach, or even add some project-list-sources hook.
>
> But how would the entries from different sources (e.g. the list file
> and your dynamic list) combined? How will they be sorted?
>
> In case we can't come up with a generic way, we could simply add a
> generic storage abstraction (similar to xref-history-storage), which
> you would override/advise to combine the lists as needed for your
> usage.

I don't think these changes are necessary after all.  (Which is good,
because they would add a fair bit of complexity.)  This file-notify
approach I think should work perfectly for a wide variety of uses.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]


Add project-watch to discover projects with file-notify

Projects can be created outside of Emacs, but users might want to be
able to switch to them with project-switch-project immediately.  This
function supports that.  If a user calls (project-watch "~/src" 1) then
any projects under ~/src will be discovered automatically immediately
after their creation.

I have strived to make this function usable for a wide set of use-cases,
I think it will be a useful addition to project.el.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-project-watch-to-discover-projects-with-file-not.patch --]
[-- Type: text/x-patch, Size: 3362 bytes --]

From b283eb7eec428b8b0027a28f8c5f547360386b80 Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@janestreet.com>
Date: Tue, 27 Jun 2023 15:25:02 -0400
Subject: [PATCH] Add project-watch to discover projects with file-notify

Projects can be created outside of Emacs, but users might want to be
able to switch to them with project-switch-project immediately.  This
API supports that.  If a user calls (project-watch "~/src" 1) then any
projects under ~/src will be discovered automatically immediately
after their creation.

* lisp/progmodes/project.el (project-check-project)
(project--watch-cb-children, project--watch-cb-this)
(project--file-notify-watch, project-watch): Add. (bug#63870)
---
 lisp/progmodes/project.el | 51 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 56c524bcab5..ddb033d50f9 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1884,5 +1884,56 @@ project-switch-project
     (let ((project-current-directory-override dir))
       (call-interactively command))))
 
+(defun project-check-project (dir)
+  "If there's a project at DIR, remember it; otherwise, forget it.
+
+Return the found project, if any."
+  (let ((pr (project--find-in-directory dir)))
+    (if pr (project-remember-project pr)
+      (project-forget-project (file-name-as-directory dir)))
+    pr))
+
+(defun project--watch-cb-children (recursive predicate event)
+  (unless (eq (cl-second event) 'stopped)
+    (dolist (file (cddr event))
+      (condition-case _ (project-watch file recursive predicate)
+        ((file-error file-notify-error))))))
+
+(defun project--watch-cb-this (dir event)
+  (unless (eq (cl-second event) 'stopped)
+    (when (project-check-project dir)
+      (file-notify-rm-watch (cl-first event)))))
+
+(defun project--file-notify-watch (dir callback &optional init)
+  "Like `file-notify-add-watch' but also calls CALLBACK immediately."
+  (let ((watch (file-notify-add-watch dir '(change) callback)))
+    (funcall callback (append (list watch 'started) init))))
+
+;;;###autoload
+(defun project-watch (dir &optional recursive predicate)
+  "Watch DIR until it becomes a project.
+
+We stop watching DIR once it becomes a project.
+
+If RECURSIVE is an integer greater than 0, we'll also run
+`project-watch' on directories which appear inside DIR,
+passing (1- RECURSIVE) as RECURSIVE.  To achieve this, we'll
+continue watching DIR even if it becomes a project.  This can be
+expensive, so it's better to pass small values of RECURSIVE, like
+1 or 2.
+
+If PREDICATE is non-nil, it should be a function which will be
+called with two arguments, the value of RECURSIVE and a
+directory.  Only directories for which PREDICATE returns non-nil
+will be watched for being a project."
+  (setq predicate (or predicate (lambda (_recursive dir) t)))
+  (setq recursive (or recursive 0))
+  (when (and (funcall predicate recursive dir) (file-directory-p dir))
+    (project--file-notify-watch dir (apply-partially #'project--watch-cb-this dir)))
+  (when (> recursive 0)
+    (project--file-notify-watch
+     dir (apply-partially #'project--watch-cb-children (1- recursive) predicate)
+     (directory-files dir 'full directory-files-no-dot-files-regexp))))
+
 (provide 'project)
 ;;; project.el ends here
-- 
2.39.3

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Tue, 27 Jun 2023 15:27:30 -0400
> 
> +(defun project-check-project (dir)
> +  "If there's a project at DIR, remember it; otherwise, forget it.
> +
> +Return the found project, if any."
> +  (let ((pr (project--find-in-directory dir)))
> +    (if pr (project-remember-project pr)
> +      (project-forget-project (file-name-as-directory dir)))
> +    pr))
> +
> +(defun project--watch-cb-children (recursive predicate event)
> +  (unless (eq (cl-second event) 'stopped)
> +    (dolist (file (cddr event))
> +      (condition-case _ (project-watch file recursive predicate)
> +        ((file-error file-notify-error))))))
> +
> +(defun project--watch-cb-this (dir event)
> +  (unless (eq (cl-second event) 'stopped)
> +    (when (project-check-project dir)
> +      (file-notify-rm-watch (cl-first event)))))
> +
> +(defun project--file-notify-watch (dir callback &optional init)
> +  "Like `file-notify-add-watch' but also calls CALLBACK immediately."
> +  (let ((watch (file-notify-add-watch dir '(change) callback)))
> +    (funcall callback (append (list watch 'started) init))))

Beware of watching a tree recursively: file notifications are not very
scalable, for more than one reason.  For example, the inotify backend
consumes a file descriptor and a slot in the descriptor set monitored
by pselect per each file/directory you watch.  And watching many
directories can overwhelm Emacs if some program (even unrelated to
Emacs) performs many file operations in that directory; VCS programs
are notorious in this regard, e.g., when you update from upstream.

> +(defun project-watch (dir &optional recursive predicate)
> +  "Watch DIR until it becomes a project.
> +
> +We stop watching DIR once it becomes a project.

This never explains what it means for a directory to "become a
project".  It should, because this doc string begs that question.

> +If RECURSIVE is an integer greater than 0, we'll also run
> +`project-watch' on directories which appear inside DIR,
> +passing (1- RECURSIVE) as RECURSIVE.  To achieve this, we'll
> +continue watching DIR even if it becomes a project.  This can be
> +expensive, so it's better to pass small values of RECURSIVE, like
> +1 or 2.

Are you sure this feature justifies the risks?  When would someone
want to use it, while simultaneously limiting the value of RECURSIVE
to some small integer?  (And what is considered "small" for these
purposes?)

Thanks.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Tue, 27 Jun 2023 15:27:30 -0400
>> 
>> +(defun project-check-project (dir)
>> +  "If there's a project at DIR, remember it; otherwise, forget it.
>> +
>> +Return the found project, if any."
>> +  (let ((pr (project--find-in-directory dir)))
>> +    (if pr (project-remember-project pr)
>> +      (project-forget-project (file-name-as-directory dir)))
>> +    pr))
>> +
>> +(defun project--watch-cb-children (recursive predicate event)
>> +  (unless (eq (cl-second event) 'stopped)
>> +    (dolist (file (cddr event))
>> +      (condition-case _ (project-watch file recursive predicate)
>> +        ((file-error file-notify-error))))))
>> +
>> +(defun project--watch-cb-this (dir event)
>> +  (unless (eq (cl-second event) 'stopped)
>> +    (when (project-check-project dir)
>> +      (file-notify-rm-watch (cl-first event)))))
>> +
>> +(defun project--file-notify-watch (dir callback &optional init)
>> +  "Like `file-notify-add-watch' but also calls CALLBACK immediately."
>> +  (let ((watch (file-notify-add-watch dir '(change) callback)))
>> +    (funcall callback (append (list watch 'started) init))))
>
> Beware of watching a tree recursively: file notifications are not very
> scalable, for more than one reason.  For example, the inotify backend
> consumes a file descriptor and a slot in the descriptor set monitored
> by pselect per each file/directory you watch.  And watching many
> directories can overwhelm Emacs if some program (even unrelated to
> Emacs) performs many file operations in that directory; VCS programs
> are notorious in this regard, e.g., when you update from upstream.

Absolutely.  I am trying to be careful about this: project-watch
shouldn't create watches on VCS directories.

>> +(defun project-watch (dir &optional recursive predicate)
>> +  "Watch DIR until it becomes a project.
>> +
>> +We stop watching DIR once it becomes a project.
>
> This never explains what it means for a directory to "become a
> project".  It should, because this doc string begs that question.

A directory "becomes a project" once some function on
project-find-functions returns non-nil for it.  I'll include this in the
docstring of the next version of the patch.

>> +If RECURSIVE is an integer greater than 0, we'll also run
>> +`project-watch' on directories which appear inside DIR,
>> +passing (1- RECURSIVE) as RECURSIVE.  To achieve this, we'll
>> +continue watching DIR even if it becomes a project.  This can be
>> +expensive, so it's better to pass small values of RECURSIVE, like
>> +1 or 2.
>
> Are you sure this feature justifies the risks?  When would someone
> want to use it, while simultaneously limiting the value of RECURSIVE
> to some small integer?  (And what is considered "small" for these
> purposes?)

Imagine, for example, that a user has a directory ~/src.  They make all
their VCS clones directly under ~/src: ~/src/emacs, ~/src/glibc, etc.
And when they work on a new project, they create that new clone under
~/src.

If the user wanted all these VCS clones to show up in Emacs as soon as
they're made, they could run (project-watch "~/src" 1).  This would
create a watch on ~/src, which would create watches on new empty
directories under ~/src (e.g. ~/src/gdb); the watch on ~/src/gdb would
stop if and when ~/src/gdb becomes a project (as defined above).

So in the steady state, if ~/src contains only projects, Emacs would run
exactly one watch, the one on ~/src.  This is definitely okay.

If, instead, ~/src has a two-level structure, where ~/src/emacs is not
itself a clone but instead contains a clone for each branch,
e.g. ~/src/emacs/emacs-29 and ~/src/emacs/trunk, then a user might run
(project-watch "~/src" 2).  Then in the steady state there would be one
watch on ~/src and one watch on each subdirectory of ~/src,
e.g. ~/src/emacs.  (This is the setup I personally have.)

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 63870@debbugs.gnu.org
> Date: Wed, 28 Jun 2023 08:05:23 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Beware of watching a tree recursively: file notifications are not very
> > scalable, for more than one reason.  For example, the inotify backend
> > consumes a file descriptor and a slot in the descriptor set monitored
> > by pselect per each file/directory you watch.  And watching many
> > directories can overwhelm Emacs if some program (even unrelated to
> > Emacs) performs many file operations in that directory; VCS programs
> > are notorious in this regard, e.g., when you update from upstream.
> 
> Absolutely.  I am trying to be careful about this: project-watch
> shouldn't create watches on VCS directories.

But below you explicitly give an example where it will.  And given the
fact that the majority of project.el projects use VCS as its backend,
I'd say we are already there...

> > Are you sure this feature justifies the risks?  When would someone
> > want to use it, while simultaneously limiting the value of RECURSIVE
> > to some small integer?  (And what is considered "small" for these
> > purposes?)
> 
> Imagine, for example, that a user has a directory ~/src.  They make all
> their VCS clones directly under ~/src: ~/src/emacs, ~/src/glibc, etc.
> And when they work on a new project, they create that new clone under
> ~/src.
> 
> If the user wanted all these VCS clones to show up in Emacs as soon as
> they're made, they could run (project-watch "~/src" 1).  This would
> create a watch on ~/src, which would create watches on new empty
> directories under ~/src (e.g. ~/src/gdb); the watch on ~/src/gdb would
> stop if and when ~/src/gdb becomes a project (as defined above).
> 
> So in the steady state, if ~/src contains only projects, Emacs would run
> exactly one watch, the one on ~/src.  This is definitely okay.
> 
> If, instead, ~/src has a two-level structure, where ~/src/emacs is not
> itself a clone but instead contains a clone for each branch,
> e.g. ~/src/emacs/emacs-29 and ~/src/emacs/trunk, then a user might run
> (project-watch "~/src" 2).  Then in the steady state there would be one
> watch on ~/src and one watch on each subdirectory of ~/src,
> e.g. ~/src/emacs.  (This is the setup I personally have.)

If you want to support one or two levels of recursion, let's support
just that and remove the too-general RECURSIVE argument.  If you think
there might be important use cases where there's more than one or two
levels of recursion, please describe them.

Once again, this is dangerous; users could easily shoot themselves in
the foot, because not many are aware of the pitfall of using file
notifications for many directories.  It makes no sense to warn against
something and at the same time let callers easily stumble upon that.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: 63870@debbugs.gnu.org
>> Date: Wed, 28 Jun 2023 08:05:23 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Beware of watching a tree recursively: file notifications are not very
>> > scalable, for more than one reason.  For example, the inotify backend
>> > consumes a file descriptor and a slot in the descriptor set monitored
>> > by pselect per each file/directory you watch.  And watching many
>> > directories can overwhelm Emacs if some program (even unrelated to
>> > Emacs) performs many file operations in that directory; VCS programs
>> > are notorious in this regard, e.g., when you update from upstream.
>> 
>> Absolutely.  I am trying to be careful about this: project-watch
>> shouldn't create watches on VCS directories.
>
> But below you explicitly give an example where it will.  And given the
> fact that the majority of project.el projects use VCS as its backend,
> I'd say we are already there...

No: the watch on a directory is removed once the directory becomes a VCS
directory.

>> > Are you sure this feature justifies the risks?  When would someone
>> > want to use it, while simultaneously limiting the value of RECURSIVE
>> > to some small integer?  (And what is considered "small" for these
>> > purposes?)
>> 
>> Imagine, for example, that a user has a directory ~/src.  They make all
>> their VCS clones directly under ~/src: ~/src/emacs, ~/src/glibc, etc.
>> And when they work on a new project, they create that new clone under
>> ~/src.
>> 
>> If the user wanted all these VCS clones to show up in Emacs as soon as
>> they're made, they could run (project-watch "~/src" 1).  This would
>> create a watch on ~/src, which would create watches on new empty
>> directories under ~/src (e.g. ~/src/gdb); the watch on ~/src/gdb would
>> stop if and when ~/src/gdb becomes a project (as defined above).
>> 
>> So in the steady state, if ~/src contains only projects, Emacs would run
>> exactly one watch, the one on ~/src.  This is definitely okay.
>> 
>> If, instead, ~/src has a two-level structure, where ~/src/emacs is not
>> itself a clone but instead contains a clone for each branch,
>> e.g. ~/src/emacs/emacs-29 and ~/src/emacs/trunk, then a user might run
>> (project-watch "~/src" 2).  Then in the steady state there would be one
>> watch on ~/src and one watch on each subdirectory of ~/src,
>> e.g. ~/src/emacs.  (This is the setup I personally have.)
>
> If you want to support one or two levels of recursion, let's support
> just that and remove the too-general RECURSIVE argument.  If you think
> there might be important use cases where there's more than one or two
> levels of recursion, please describe them.

Hm, well, I assume some users might use even more structure than this;
for example, some might have ~/src/gnu/emacs/emacs-29.  Then they'd want
3 levels of recursion.

> Once again, this is dangerous; users could easily shoot themselves in
> the foot, because not many are aware of the pitfall of using file
> notifications for many directories.  It makes no sense to warn against
> something and at the same time let callers easily stumble upon that.

I agree with that, I suppose.  Personally I would be fine with a
mandatory 1 or 2 levels of recursion, since I only need 2.  Do you have
a suggestion for what that interface could look like?  It feels a bit
awkward...

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 63870@debbugs.gnu.org
> Date: Wed, 28 Jun 2023 08:37:58 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Absolutely.  I am trying to be careful about this: project-watch
> >> shouldn't create watches on VCS directories.
> >
> > But below you explicitly give an example where it will.  And given the
> > fact that the majority of project.el projects use VCS as its backend,
> > I'd say we are already there...
> 
> No: the watch on a directory is removed once the directory becomes a VCS
> directory.

No, AFAIU the watch is removed once there is a project in the
directory.  But a VCS can work on a directory regardles of any project
creation.

> > Once again, this is dangerous; users could easily shoot themselves in
> > the foot, because not many are aware of the pitfall of using file
> > notifications for many directories.  It makes no sense to warn against
> > something and at the same time let callers easily stumble upon that.
> 
> I agree with that, I suppose.  Personally I would be fine with a
> mandatory 1 or 2 levels of recursion, since I only need 2.  Do you have
> a suggestion for what that interface could look like?  It feels a bit
> awkward...

I'd actually begin by not providing even 1 level.  Let the callers
call this new function  explicitly for every directory which they want
watching.  If someone ever complains that this is somehow inconvenient
(although I don't see why: directory-files is simple to use), then we
could consider extending the API.

But that's MO; please wait for Dmitry to chime in.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Dmitry Gutov]

On 28/06/2023 15:56, Eli Zaretskii wrote:
>>> Once again, this is dangerous; users could easily shoot themselves in
>>> the foot, because not many are aware of the pitfall of using file
>>> notifications for many directories.  It makes no sense to warn against
>>> something and at the same time let callers easily stumble upon that.
>> I agree with that, I suppose.  Personally I would be fine with a
>> mandatory 1 or 2 levels of recursion, since I only need 2.  Do you have
>> a suggestion for what that interface could look like?  It feels a bit
>> awkward...
> I'd actually begin by not providing even 1 level.  Let the callers
> call this new function  explicitly for every directory which they want
> watching.  If someone ever complains that this is somehow inconvenient
> (although I don't see why: directory-files is simple to use), then we
> could consider extending the API.

That sounds about right. But I might go a little further in this 
reasoning... (*)

> But that's MO; please wait for Dmitry to chime in.

[ Sorry for the late response, I'm still uncertain about this patch. ]

(*) ... and ask whether this functionality makes sense built-in.

I appreciate that it's succinct, documented and doesn't take a lot of 
space. But would we say that it covers a significantly general use case? 
Do we know many other developers who would appreciate it? Do a lot of 
devs at Jane Street use Emacs and this same workflow? Should we ask 
people somewhere (emacs-devel/Reddit/etc) whether they will find it useful?

If it's just for one user at this point, then it shouldn't be difficult 
to maintain this code inside the init dir.

Here's also some alternative I could potentially suggest: if you have 
some code which checks out new branches for development, or projects to 
start work on, and it's written in Elisp too, could it just call 
project-remember-project at the end? That would circumvent the need for 
using file watches altogether.

Or if we do add this to project.el, we should try to make it safe for an 
average user even with a different directory structure. Suppose they 
have a dir D which they call project-watch on, and then they copy a big 
non-project directory inside. That should trigger many filenotify 
events, and since no search would result in success, I suppose the watch 
stays on, and every directory gets scanned up until the root. So an 
easy-to-enable recursive behavior seems dangerous for this case.

Needless to say, the user could call this function, spend time on other 
stuff, forget, and then get surprised by things taking longer than expected.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 28/06/2023 15:56, Eli Zaretskii wrote:
>>>> Once again, this is dangerous; users could easily shoot themselves in
>>>> the foot, because not many are aware of the pitfall of using file
>>>> notifications for many directories.  It makes no sense to warn against
>>>> something and at the same time let callers easily stumble upon that.
>>> I agree with that, I suppose.  Personally I would be fine with a
>>> mandatory 1 or 2 levels of recursion, since I only need 2.  Do you have
>>> a suggestion for what that interface could look like?  It feels a bit
>>> awkward...
>> I'd actually begin by not providing even 1 level.  Let the callers
>> call this new function  explicitly for every directory which they want
>> watching.  If someone ever complains that this is somehow inconvenient
>> (although I don't see why: directory-files is simple to use), then we
>> could consider extending the API.
>
> That sounds about right. But I might go a little further in this
> reasoning... (*)
>
>> But that's MO; please wait for Dmitry to chime in.
>
> [ Sorry for the late response, I'm still uncertain about this patch. ]
>
> (*) ... and ask whether this functionality makes sense built-in.
>
> I appreciate that it's succinct, documented and doesn't take a lot of
> space. But would we say that it covers a significantly general use
> case? Do we know many other developers who would appreciate it? Do a
> lot of devs at Jane Street use Emacs and this same workflow? Should we
> ask people somewhere (emacs-devel/Reddit/etc) whether they will find
> it useful?

It's something we'd use a lot, but I have no problem keeping it locally
for now.  I agree it has some dangers, maybe I'll come up with a good
way to make it safe for the casual user.

> If it's just for one user at this point, then it shouldn't be
> difficult to maintain this code inside the init dir.
>
> Here's also some alternative I could potentially suggest: if you have
> some code which checks out new branches for development, or projects
> to start work on, and it's written in Elisp too, could it just call
> project-remember-project at the end? That would circumvent the need
> for using file watches altogether.

That works great for stuff written in Elisp too, but alas, there are
command line tools to check out branches for development, etc, and I
want them to update Emacs too.

> Or if we do add this to project.el, we should try to make it safe for
> an average user even with a different directory structure. Suppose
> they have a dir D which they call project-watch on, and then they copy
> a big non-project directory inside. That should trigger many
> filenotify events, and since no search would result in success, I
> suppose the watch stays on, and every directory gets scanned up until
> the root. So an easy-to-enable recursive behavior seems dangerous for
> this case.
>
> Needless to say, the user could call this function, spend time on
> other stuff, forget, and then get surprised by things taking longer
> than expected.

Definitely.  I'll think about whether there's a way to avoid such
footguns...

(Maybe I'll revisit this after writing some scripts like this for Emacs
development - although those will definitely only be in Elisp, so it
would be kind of moot)

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Juri Linkov]

> (Maybe I'll revisit this after writing some scripts like this for Emacs
> development - although those will definitely only be in Elisp, so it
> would be kind of moot)

Why can't project--ensure-read-project-list scan a list of known roots
to find a new project?  I.e. the reverse of project-forget-projects-under.

Or like there is an item "... (choose a dir)", another item could be
"*Rescan*" that will try to find a new project under known roots.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Dmitry Gutov]

On 18/07/2023 20:41, Juri Linkov wrote:
>> (Maybe I'll revisit this after writing some scripts like this for Emacs
>> development - although those will definitely only be in Elisp, so it
>> would be kind of moot)
> Why can't project--ensure-read-project-list scan a list of known roots
> to find a new project?  I.e. the reverse of project-forget-projects-under.
> 
> Or like there is an item "... (choose a dir)", another item could be
> "*Rescan*" that will try to find a new project under known roots.

We have an interactive command called 'project-remember-projects-under', 
when you want to trigger such a scan.

But as for why we don't do that automatically? Performance, for 
instance. Tree traversal, coupled with arbitrary logic for finding 
projects, can be too costly.

bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Dmitry Gutov]

On 18/07/2023 19:28, Spencer Baugh wrote:

>>> But that's MO; please wait for Dmitry to chime in.
>>
>> [ Sorry for the late response, I'm still uncertain about this patch. ]
>>
>> (*) ... and ask whether this functionality makes sense built-in.
>>
>> I appreciate that it's succinct, documented and doesn't take a lot of
>> space. But would we say that it covers a significantly general use
>> case? Do we know many other developers who would appreciate it? Do a
>> lot of devs at Jane Street use Emacs and this same workflow? Should we
>> ask people somewhere (emacs-devel/Reddit/etc) whether they will find
>> it useful?
> 
> It's something we'd use a lot, but I have no problem keeping it locally
> for now.  I agree it has some dangers, maybe I'll come up with a good
> way to make it safe for the casual user.

Thank you.

>> If it's just for one user at this point, then it shouldn't be
>> difficult to maintain this code inside the init dir.
>>
>> Here's also some alternative I could potentially suggest: if you have
>> some code which checks out new branches for development, or projects
>> to start work on, and it's written in Elisp too, could it just call
>> project-remember-project at the end? That would circumvent the need
>> for using file watches altogether.
> 
> That works great for stuff written in Elisp too, but alas, there are
> command line tools to check out branches for development, etc, and I
> want them to update Emacs too.

Just something to consider: you could invoke those shell scripts from 
Emacs Lisp, and then they finish, the Lisp program could do some 
additional steps, like adding projects to the list. Or removing -- in 
the reverse case.