bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Eli Zaretskii <eliz@gnu.org>]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 63870@debbugs.gnu.org
> Date: Wed, 28 Jun 2023 08:05:23 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Beware of watching a tree recursively: file notifications are not very
> > scalable, for more than one reason.  For example, the inotify backend
> > consumes a file descriptor and a slot in the descriptor set monitored
> > by pselect per each file/directory you watch.  And watching many
> > directories can overwhelm Emacs if some program (even unrelated to
> > Emacs) performs many file operations in that directory; VCS programs
> > are notorious in this regard, e.g., when you update from upstream.
> 
> Absolutely.  I am trying to be careful about this: project-watch
> shouldn't create watches on VCS directories.

But below you explicitly give an example where it will.  And given the
fact that the majority of project.el projects use VCS as its backend,
I'd say we are already there...

> > Are you sure this feature justifies the risks?  When would someone
> > want to use it, while simultaneously limiting the value of RECURSIVE
> > to some small integer?  (And what is considered "small" for these
> > purposes?)
> 
> Imagine, for example, that a user has a directory ~/src.  They make all
> their VCS clones directly under ~/src: ~/src/emacs, ~/src/glibc, etc.
> And when they work on a new project, they create that new clone under
> ~/src.
> 
> If the user wanted all these VCS clones to show up in Emacs as soon as
> they're made, they could run (project-watch "~/src" 1).  This would
> create a watch on ~/src, which would create watches on new empty
> directories under ~/src (e.g. ~/src/gdb); the watch on ~/src/gdb would
> stop if and when ~/src/gdb becomes a project (as defined above).
> 
> So in the steady state, if ~/src contains only projects, Emacs would run
> exactly one watch, the one on ~/src.  This is definitely okay.
> 
> If, instead, ~/src has a two-level structure, where ~/src/emacs is not
> itself a clone but instead contains a clone for each branch,
> e.g. ~/src/emacs/emacs-29 and ~/src/emacs/trunk, then a user might run
> (project-watch "~/src" 2).  Then in the steady state there would be one
> watch on ~/src and one watch on each subdirectory of ~/src,
> e.g. ~/src/emacs.  (This is the setup I personally have.)

If you want to support one or two levels of recursion, let's support
just that and remove the too-general RECURSIVE argument.  If you think
there might be important use cases where there's more than one or two
levels of recursion, please describe them.

Once again, this is dangerous; users could easily shoot themselves in
the foot, because not many are aware of the pitfall of using file
notifications for many directories.  It makes no sense to warn against
something and at the same time let callers easily stumble upon that.