bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh <sbaugh@janestreet.com>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Tue, 27 Jun 2023 15:27:30 -0400
>> 
>> +(defun project-check-project (dir)
>> +  "If there's a project at DIR, remember it; otherwise, forget it.
>> +
>> +Return the found project, if any."
>> +  (let ((pr (project--find-in-directory dir)))
>> +    (if pr (project-remember-project pr)
>> +      (project-forget-project (file-name-as-directory dir)))
>> +    pr))
>> +
>> +(defun project--watch-cb-children (recursive predicate event)
>> +  (unless (eq (cl-second event) 'stopped)
>> +    (dolist (file (cddr event))
>> +      (condition-case _ (project-watch file recursive predicate)
>> +        ((file-error file-notify-error))))))
>> +
>> +(defun project--watch-cb-this (dir event)
>> +  (unless (eq (cl-second event) 'stopped)
>> +    (when (project-check-project dir)
>> +      (file-notify-rm-watch (cl-first event)))))
>> +
>> +(defun project--file-notify-watch (dir callback &optional init)
>> +  "Like `file-notify-add-watch' but also calls CALLBACK immediately."
>> +  (let ((watch (file-notify-add-watch dir '(change) callback)))
>> +    (funcall callback (append (list watch 'started) init))))
>
> Beware of watching a tree recursively: file notifications are not very
> scalable, for more than one reason.  For example, the inotify backend
> consumes a file descriptor and a slot in the descriptor set monitored
> by pselect per each file/directory you watch.  And watching many
> directories can overwhelm Emacs if some program (even unrelated to
> Emacs) performs many file operations in that directory; VCS programs
> are notorious in this regard, e.g., when you update from upstream.

Absolutely.  I am trying to be careful about this: project-watch
shouldn't create watches on VCS directories.

>> +(defun project-watch (dir &optional recursive predicate)
>> +  "Watch DIR until it becomes a project.
>> +
>> +We stop watching DIR once it becomes a project.
>
> This never explains what it means for a directory to "become a
> project".  It should, because this doc string begs that question.

A directory "becomes a project" once some function on
project-find-functions returns non-nil for it.  I'll include this in the
docstring of the next version of the patch.

>> +If RECURSIVE is an integer greater than 0, we'll also run
>> +`project-watch' on directories which appear inside DIR,
>> +passing (1- RECURSIVE) as RECURSIVE.  To achieve this, we'll
>> +continue watching DIR even if it becomes a project.  This can be
>> +expensive, so it's better to pass small values of RECURSIVE, like
>> +1 or 2.
>
> Are you sure this feature justifies the risks?  When would someone
> want to use it, while simultaneously limiting the value of RECURSIVE
> to some small integer?  (And what is considered "small" for these
> purposes?)

Imagine, for example, that a user has a directory ~/src.  They make all
their VCS clones directly under ~/src: ~/src/emacs, ~/src/glibc, etc.
And when they work on a new project, they create that new clone under
~/src.

If the user wanted all these VCS clones to show up in Emacs as soon as
they're made, they could run (project-watch "~/src" 1).  This would
create a watch on ~/src, which would create watches on new empty
directories under ~/src (e.g. ~/src/gdb); the watch on ~/src/gdb would
stop if and when ~/src/gdb becomes a project (as defined above).

So in the steady state, if ~/src contains only projects, Emacs would run
exactly one watch, the one on ~/src.  This is definitely okay.

If, instead, ~/src has a two-level structure, where ~/src/emacs is not
itself a clone but instead contains a clone for each branch,
e.g. ~/src/emacs/emacs-29 and ~/src/emacs/trunk, then a user might run
(project-watch "~/src" 2).  Then in the steady state there would be one
watch on ~/src and one watch on each subdirectory of ~/src,
e.g. ~/src/emacs.  (This is the setup I personally have.)