bug#50297: 28.0.50; Aggregate project functions for project.el

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]


The following patch introduces a few functions for aggregate project
maintenance:

- project-find-projects-under
  Select a directory with projects to index all at once.
- project-remove-zombie-projects
  Check if all known projects still exist and remove those
  that don't anymore
- project-remove-projects-under
  Remove all projects in a directory (inverse of
  project-find-projects-under).

Especially the last two are useful to maintain a clean project list
without having to manually remove every project one by one.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-aggregate-project-discovery-and-maintenance-func.patch --]
[-- Type: text/x-diff, Size: 3036 bytes --]

From 3431a9123753d769f10621d2f5f6ef72ab0e2f3a Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Tue, 31 Aug 2021 14:12:13 +0200
Subject: [PATCH] Add aggregate project discovery and maintenance functions

* project.el (project-remember-project): Add optional no-write argument
(project-find-projects-under): Add command
(project-remove-zombie-projects): Add command
(project-remove-known-projects): Add command
---
 lisp/progmodes/project.el | 37 ++++++++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index ae9bf03571..2f251393e2 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1261,9 +1261,10 @@ project--write-project-list
       (write-region nil nil filename nil 'silent))))
 
 ;;;###autoload
-(defun project-remember-project (pr)
+(defun project-remember-project (pr &optional no-write)
   "Add project PR to the front of the project list.
-Save the result in `project-list-file' if the list of projects has changed."
+Save the result in `project-list-file' if the list of projects
+has changed, and NO-WRITE is nil."
   (project--ensure-read-project-list)
   (let ((dir (project-root pr)))
     (unless (equal (caar project--list) dir)
@@ -1271,7 +1272,8 @@ project-remember-project
         (when (equal dir (car ent))
           (setq project--list (delq ent project--list))))
       (push (list dir) project--list)
-      (project--write-project-list))))
+      (unless no-write
+        (project--write-project-list)))))
 
 (defun project--remove-from-project-list (project-root report-message)
   "Remove directory PROJECT-ROOT of a missing project from the project list.
@@ -1325,6 +1327,35 @@ project-execute-extended-command
   (let ((default-directory (project-root (project-current t))))
     (call-interactively #'execute-extended-command)))
 
+(defun project-find-projects-under (dir)
+  "Index all projects below a directory DIR."
+  (interactive "DDirectory: ")
+  (let ((count 0))
+    (dolist (subdir (directory-files dir t nil t))
+      (when-let (pr (project--find-in-directory subdir))
+        (project-remember-project pr t)
+        (message "Found %s..." (project-root pr))
+        (setq count (1+ count))))
+    (if (zerop count)
+        (message "No projects found")
+      (project--write-project-list)
+      (message "%d project%s found"
+               count (if (= count 1) "" "s")))))
+
+(defun project-remove-zombie-projects ()
+  "Remove all known projects that don't exist any more."
+  (interactive)
+  (dolist (proj (project-known-project-roots))
+    (unless (file-exists-p proj)
+      (project-remove-known-project proj))))
+
+(defun project-remove-known-projects (dir)
+  "Remove all known projects below a directory DIR."
+  (interactive "DDirectory: ")
+  (dolist (proj (project-known-project-roots))
+    (when (file-in-directory-p proj dir)
+      (project-remove-known-project proj))))
+
 \f
 ;;; Project switching
 
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 6809 bytes --]




In GNU Emacs 28.0.50 (build 7, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw scroll bars)
 of 2021-08-26 built on icterid
Repository revision: b74afe63d02f126127d9211c0d1c1dc2cf5dd5bb
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure LDFLAGS=-flto 'CFLAGS=-O2 -march=native -mtune=native -pipe'
 --with-native-compiler'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11
XDBE XIM XPM LUCID ZLIB

Important settings:
  value of $EMACSLOADPATH: 
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Git-Log-View

Minor modes in effect:
  TeX-PDF-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  icomplete-mode: t
  rcirc-track-minor-mode: t
  display-time-mode: t
  winner-mode: t
  windmove-mode: t
  electric-pair-mode: t
  recentf-mode: t
  save-place-mode: t
  savehist-mode: t
  show-paren-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/philip/.config/emacs/elpa/transient-0.3.6/transient hides /home/philip/Code/src/emacs/lisp/transient
~/.config/emacs/site-lisp/autoload hides /home/philip/Code/src/emacs/lisp/emacs-lisp/autoload

Features:
(shadow emacsbug tramp-archive tramp-gvfs tramp-cache zeroconf tramp
tramp-loaddefs trampver tramp-integration files-x tramp-compat ls-lisp
flymake-cc macrostep-c cmacexp macrostep preview tex-buf tex-fold
reftex-dcr reftex-auc reftex reftex-loaddefs reftex-vars font-latex
latex latex-flymake tex-ispell tex-style tex texmathp tex-mode latexenc
apropos eieio-opt speedbar ezimage dframe shortdoc vc-annotate
help-at-pt gnus-fun cl-print debug backtrace pulse find-func rect
markdown-mode shell-command+ rng-xsd xsd-regexp rng-cmpct rng-nxml
rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util
rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-enc
xmltok xref find-dired grep mhtml-mode css-mode smie eww xdg url-queue
mm-url color js cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs sgml-mode facemenu whitespace
make-mode dired-aux ffap avy magit-extras bug-reference face-remap
magit-submodule magit-obsolete magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log which-func imenu
magit-diff git-commit log-edit add-log magit-core magit-autorevert
autorevert filenotify magit-margin magit-transient magit-process
with-editor term ehelp eshell esh-cmd esh-ext esh-opt esh-proc esh-io
esh-arg esh-module esh-groups esh-util shell pcomplete server magit-mode
transient format-spec magit-git magit-section magit-utils dash vc-fossil
vc-mtn vc-hg vc-git vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs icomplete
project memory-report char-fold misearch multi-isearch mailalias
bbdb-pgp url-http url-gw url-cache url-auth cus-edit pp cus-start
finder-inf bbdb-message autocrypt-message smerge-mode diff-mode
jka-compr mule-util smiley gnus-cite flow-fill mm-archive mail-extr
gnus-async gnus-bcklg qp sort gnus-ml disp-table autocrypt-gnus
autocrypt nndraft nnmh epa-file gnutls network-stream nsm nnmaildir
nnfolder vc-backup log-view pcvs-util vc vc-dispatcher diff time-stamp
bbdb-gnus bbdb-mua bbdb-com crm nnnil gnus-agent gnus-srvr gnus-score
score-mode nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime
smime dig nntp gnus-cache gnus-sum shr kinsoku svg dom gnus-group
gnus-undo gnus-start gnus-dbus dbus xml gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range message rmc
puny rfc822 mml mml-sec epa mm-decode mm-bodies mm-encode mailabbrev
gmm-utils mailheader gnus-win modus-vivendi-theme paredit checkdoc
flymake-proc flymake warnings thingatpt flyspell ispell noutline outline
easy-mmode gnus-dired dired-x dired dired-loaddefs rcirc parse-time
iso8601 rx time bbdb bbdb-site timezone sendmail gnus nnheader gnus-util
rmail rmail-loaddefs time-date mail-utils hippie-exp winner windmove
elec-pair recentf tree-widget wid-edit saveplace savehist paren
modus-operandi-theme modus-themes holidays hol-loaddefs cal-menu
calendar cal-loaddefs cus-load setup load compile text-property-search
comint ansi-color autoload lisp-mnt mail-parse rfc2231 rfc2047 rfc2045
mm-util ietf-drums mail-prsvr tex-site geiser-impl help-fns radix-tree
geiser-custom geiser-base ring slime-autoloads info package let-alist
derived edmacro kmacro pcase cl-extra help-mode browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util mailcap url-handlers url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs password-cache json map url-vars seq
byte-opt gv bytecomp byte-compile cconv epg epg-config subr-x
cl-loaddefs cl-lib iso-transl tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar
mouse jit-lock font-lock syntax font-core term/tty-colors frame
minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite charscript charprop case-table epa-hook jka-cmpr-hook help
simple abbrev obarray cl-preloaded nadvice button loaddefs faces
cus-face macroexp files window text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo x-toolkit x multi-tty make-network-process
emacs)

Memory information:
((conses 16 1167755 519125)
 (symbols 48 100287 1576)
 (strings 32 294014 1164412)
 (string-bytes 1 10068553)
 (vectors 16 85242)
 (vector-slots 8 1729184 892975)
 (floats 8 776 1974)
 (intervals 56 55886 9874)
 (buffers 992 65))

-- 
	Philip K.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

Hi!

On 31.08.2021 15:47, Philip Kaludercic wrote:
> 
> The following patch introduces a few functions for aggregate project
> maintenance:
> 
> - project-find-projects-under
>    Select a directory with projects to index all at once.

I wonder how popular this is going to be. Do you have a flat directory 
with projects which you only want scanned one time?

Another issue, is that it's not going to find nested projects (and 
project.el does support those).

Suppose we do add it, how about the name 
'project-remember-projects-under'? By analogy with 
'project-remember-project'.

Adding a new arg for the latter is fine by me either way.

> - project-remove-zombie-projects
>    Check if all known projects still exist and remove those
>    that don't anymore

Perhaps we should rename 'project-remove-known-project' to 
'project-forget-known-project'? That would make for a nice symmetry.

Then this function could be called 'project-forget-zombie-projects'.

I'm thinking about this about the slight connotation of 'remove' which 
can mean removing from disk.

Another approach would be to call this or similar code automatically 
before saving the list (and cap the number of remembered projects), but 
that comes with its own tradeoffs.

> - project-remove-projects-under
>    Remove all projects in a directory (inverse of
>    project-find-projects-under).

Similar question about popularity, but this one won't have a problem 
with semantics, at least (recursive-vs-non-recursive).

> Especially the last two are useful to maintain a clean project list
> without having to manually remove every project one by one.

What if the goal was to maintain a clean project list but minimize the 
manual management of it by the user?

Can you imagine a solution for that? What would be the downsides, 
compared to the present proposal?

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Hi!
>
> On 31.08.2021 15:47, Philip Kaludercic wrote:
>> The following patch introduces a few functions for aggregate project
>> maintenance:
>> - project-find-projects-under
>>    Select a directory with projects to index all at once.
>
> I wonder how popular this is going to be. Do you have a flat directory
> with projects which you only want scanned one time?

I clone most code into a directory, and I have seen others do so
too. That being said, it might just be something unusual in the big
picture.

> Another issue, is that it's not going to find nested projects (and
> project.el does support those).

My first implementation of the command tried to so something like that,
but it was rather slow (even if I currently only have 20 projects
checked out), and indexed a lot of projects that I wasn't interested
in. Maybe I can look into how it can be accelerated or only search for
nested projects when a prefix argument is supplied/not supplied.

> Suppose we do add it, how about the name
> 'project-remember-projects-under'? By analogy with
> 'project-remember-project'.

I like it.

> Adding a new arg for the latter is fine by me either way.
>
>> - project-remove-zombie-projects
>>    Check if all known projects still exist and remove those
>>    that don't anymore
>
> Perhaps we should rename 'project-remove-known-project' to
> 'project-forget-known-project'? That would make for a nice symmetry.
>
> Then this function could be called 'project-forget-zombie-projects'.

This also make sense. Initially I wanted to name the command that way,
but then decided to go with "remove" to keep the naming consistent.

> I'm thinking about this about the slight connotation of 'remove' which
> can mean removing from disk.
>
> Another approach would be to call this or similar code automatically
> before saving the list (and cap the number of remembered projects),
> but that comes with its own tradeoffs.

I can try it out, but I fear it might lead to annoying pauses,
especially when a project was indexed via TRAMP.

>> - project-remove-projects-under
>>    Remove all projects in a directory (inverse of
>>    project-find-projects-under).
>
> Similar question about popularity, but this one won't have a problem
> with semantics, at least (recursive-vs-non-recursive).
>
>> Especially the last two are useful to maintain a clean project list
>> without having to manually remove every project one by one.
>
> What if the goal was to maintain a clean project list but minimize the
> manual management of it by the user?
>
> Can you imagine a solution for that? What would be the downsides,
> compared to the present proposal?

I can imagine zombie projects being cleaned up automatically, but
the motivation to write project-remove-projects-under was to remove
projects that were falsely indexed.

An entirely different approach might be to implement a tabulated list
major mode to manage projects, comparable to package-list.

-- 
	Philip Kaludercic

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> An entirely different approach might be to implement a tabulated list
> major mode to manage projects, comparable to package-list.

For the sake of it, it tried it out how this might look like. It feels
clunky as of now, doesn't implement everything that it should and it
might make more sense to provide as an additional package.


[-- Attachment #2: Type: text/plain, Size: 3793 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index e420a4ccca..b438249b95 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1327,6 +1327,102 @@ project-execute-extended-command
   (let ((default-directory (project-root (project-current t))))
     (call-interactively #'execute-extended-command)))
 
+\f
+;;; Project managment
+
+(defun project-list-generate-list ()
+  "Generate a list of projects for `tabulated-list-mode'."
+  (let (entries)
+    (dolist (root (project-known-project-roots))
+      (when-let* ((proj (project--find-in-directory root))
+                  (root (project-root proj))
+                  ;; XXX: Name and Type are just to keep the buffer
+                  ;;      from looking too empty.
+                  (name (capitalize
+                         (file-name-nondirectory
+                          (directory-file-name root))))
+                  (type (if (consp proj) (format "%S" (car proj)) "??"))
+                  (data (vector name type root)))
+        (push (list root data) entries)))
+    entries))
+
+(defun project-list-select ()
+  "Select the project at point."
+  (interactive)
+  (project-switch-project (tabulated-list-get-id)))
+
+(defun project-list-mark-forget ()
+  "Mark the project at point to be forgotten."
+  (interactive)
+  (save-mark-and-excursion
+    (save-restriction
+      (narrow-to-region (region-beginning) (region-end))
+      (goto-char (point-min))
+      (while (not (eobp))
+        (tabulated-list-put-tag "F" t)))))
+
+(defun project-list-forget-zombies ()
+  "Mark the project at point to be forgotten."
+  (interactive)
+  (save-excursion
+    (goto-char (point-min))
+    (while (not (eobp))
+      (unless (file-exists-p (tabulated-list-get-id))
+        (tabulated-list-put-tag "F" t)))))
+
+(defun project-list-unmark ()
+  "Unmark the project at point."
+  (save-mark-and-excursion
+    (save-restriction
+      (narrow-to-region (region-beginning) (region-end))
+      (goto-char (point-min))
+      (while (not (eobp))
+        (tabulated-list-put-tag " " t)))))
+
+(defun project-list-execute ()
+  "Preform marked actions on the project list."
+  (interactive)
+  (let (forget-list)
+    (save-excursion
+      (goto-char (point-min))
+      (while (not (eobp))
+        (when (eq (char-after) ?F)
+          (push (tabulated-list-get-id) forget-list))
+        (forward-line)))
+    (when (yes-or-no-p (format "Forget %d projects? " (length forget-list)))
+      (mapc #'project-remove-known-project forget-list)
+      (tabulated-list-clear-all-tags)
+      (tabulated-list-print))))
+
+(defvar project-list-mode-map
+  (let ((map (make-sparse-keymap)))
+    (define-key map (kbd "RET") #'project-list-select)
+    (define-key map (kbd "f") #'project-list-mark-forget)
+    (define-key map (kbd "d") #'project-list-mark-forget)
+    (define-key map (kbd "z") #'project-list-forget-zombies)
+    (define-key map (kbd "u") #'project-list-unmark)
+    (define-key map (kbd "x") #'project-list-execute)
+    map))
+
+(define-derived-mode project-list-mode tabulated-list-mode "Project List"
+  "Major mode for browsing the list of known projects."
+  (setq tabulated-list-format [("Name" 16 t)
+                               ("Type" 4 nil)
+                               ("Path" 0 t)]
+        tabulated-list-entries #'project-list-generate-list
+        tabulated-list-padding 2)
+  (tabulated-list-init-header)
+  (tabulated-list-print))
+
+;;;###autoload
+(defun project-list-projects ()
+  "Display a list of all known projects."
+  (interactive)
+  (project--ensure-read-project-list)
+  (with-current-buffer (get-buffer-create "*Projects*")
+    (project-list-mode)
+    (pop-to-buffer-same-window (current-buffer))))
+
 \f
 ;;; Project switching
 

[-- Attachment #3: Type: text/plain, Size: 24 bytes --]


-- 
	Philip Kaludercic

bug#50297: 28.0.50; Aggregate project functions for project.el

[Juri Linkov]

>> An entirely different approach might be to implement a tabulated list
>> major mode to manage projects, comparable to package-list.
>
> For the sake of it, it tried it out how this might look like. It feels
> clunky as of now, doesn't implement everything that it should and it
> might make more sense to provide as an additional package.

This is like music players that provide an option to scan a folder for
music files, then allow to manage found files in a list.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

On 02.09.2021 16:30, Philip Kaludercic wrote:

>>> - project-find-projects-under
>>>     Select a directory with projects to index all at once.
>>
>> I wonder how popular this is going to be. Do you have a flat directory
>> with projects which you only want scanned one time?
> 
> I clone most code into a directory, and I have seen others do so
> too. That being said, it might just be something unusual in the big
> picture.

I think it's usual enough, but since we try to enable all kinds of 
practices, we should try to handle as many as feasible.

I'm mostly worried about the semantics here: we say we scan all the 
projects in said directory, but we can miss some (or many). That can be 
fixed with an edit to the docstring, of course (e.g. "directly below").

>> Another issue, is that it's not going to find nested projects (and
>> project.el does support those).
> 
> My first implementation of the command tried to so something like that,
> but it was rather slow (even if I currently only have 20 projects
> checked out), and indexed a lot of projects that I wasn't interested
> in. Maybe I can look into how it can be accelerated or only search for
> nested projects when a prefix argument is supplied/not supplied.

It might be fine as it is, behavior-wise. It never occurred to me to ask 
or search for such functionality, though, so me might want to wait for 
others to chime in (who will want to use this). I'm usually fine with 
'C-x C-f' when visiting a project the first time.

>> Suppose we do add it, how about the name
>> 'project-remember-projects-under'? By analogy with
>> 'project-remember-project'.
> 
> I like it.
> 
>> Adding a new arg for the latter is fine by me either way.
>>
>>> - project-remove-zombie-projects
>>>     Check if all known projects still exist and remove those
>>>     that don't anymore
>>
>> Perhaps we should rename 'project-remove-known-project' to
>> 'project-forget-known-project'? That would make for a nice symmetry.
>>
>> Then this function could be called 'project-forget-zombie-projects'.
> 
> This also make sense. Initially I wanted to name the command that way,
> but then decided to go with "remove" to keep the naming consistent.

We might encounter some resistance (the command has been with us for 
half a year), but let's try it, at least. We can also ask Theodor 
whether he has already used the current name, 
project-remove-known-project, in some Lisp code.

>> I'm thinking about this about the slight connotation of 'remove' which
>> can mean removing from disk.
>>
>> Another approach would be to call this or similar code automatically
>> before saving the list (and cap the number of remembered projects),
>> but that comes with its own tradeoffs.
> 
> I can try it out, but I fear it might lead to annoying pauses,
> especially when a project was indexed via TRAMP.

We could skip remote projects entirely in that function (leaving it to 
the user to 'forget' them manually), or only do it for already connected 
hosts. The latter could still take some time, though.

Do you expect project-forget-zombie-projects to be useful to remote 
hosts as well? Do you also have directories on remote hosts that contain 
many projects?

>>> Especially the last two are useful to maintain a clean project list
>>> without having to manually remove every project one by one.
>>
>> What if the goal was to maintain a clean project list but minimize the
>> manual management of it by the user?
>>
>> Can you imagine a solution for that? What would be the downsides,
>> compared to the present proposal?
> 
> I can imagine zombie projects being cleaned up automatically, but
> the motivation to write project-remove-projects-under was to remove
> projects that were falsely indexed.

Could you elaborate? Falsely meaning projects you simply don't want to 
return to? A whole directory of them?

> An entirely different approach might be to implement a tabulated list
> major mode to manage projects, comparable to package-list.

I don't object to supporting this approach, built-in or in a third-party 
package, as long as we're reasonably sure there is no low-touch, 
automatic solution we can use with a similar success for the problem in 
question.

If there's none, or if we found such, and then figured that some manual 
management on top of it can still be useful, that is fine.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

On 02.09.2021 17:45, Philip Kaludercic wrote:
> For the sake of it, it tried it out how this might look like. It feels
> clunky as of now, doesn't implement everything that it should and it
> might make more sense to provide as an additional package.

I like it as an idea, but probably wouldn't use the UI itself much. At 
this point it's up to you which of the approaches to put forward, though.

I was also thinking that the "read project" prompt itself could serve 
like such a listing which could be edited by the user on-the-fly. But 
that would require some extension of the completing-read facility.

The closest analogy is how icomplete-fido-kill (C-k) can delete a file 
if it's called during file name completion.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

On 31.08.2021 15:47, Philip Kaludercic wrote:
> The following patch introduces a few functions for aggregate project
> maintenance:
> 
> - project-find-projects-under
>    Select a directory with projects to index all at once.
> - project-remove-zombie-projects
>    Check if all known projects still exist and remove those
>    that don't anymore
> - project-remove-projects-under
>    Remove all projects in a directory (inverse of
>    project-find-projects-under).
> 
> Especially the last two are useful to maintain a clean project list
> without having to manually remove every project one by one.

OK, so I have done the rename: we now have project-forget-project.

While I'm not necessarily a fan of the tabulated list approach, the 
above list looks sensible. Let's just name them to fit the current 
scheme better:

   project-remember-projects-under
   project-forget-projects-under
   project-forget-zombie-projects

And whatever ambiguities about recursive search can be solved through 
better descriptions in docstrings.

We could also add a hook like project-after-read-list-hook, which people 
would be able to use for cleanup, e.g. adding 
project-forget-zombie-projects to it. Not sure whether to add it there 
by default, though.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 31.08.2021 15:47, Philip Kaludercic wrote:
>> The following patch introduces a few functions for aggregate project
>> maintenance:
>> - project-find-projects-under
>>    Select a directory with projects to index all at once.
>> - project-remove-zombie-projects
>>    Check if all known projects still exist and remove those
>>    that don't anymore
>> - project-remove-projects-under
>>    Remove all projects in a directory (inverse of
>>    project-find-projects-under).
>> Especially the last two are useful to maintain a clean project list
>> without having to manually remove every project one by one.
>
> OK, so I have done the rename: we now have project-forget-project.
>
> While I'm not necessarily a fan of the tabulated list approach, the
> above list looks sensible. Let's just name them to fit the current
> scheme better:
>
>   project-remember-projects-under
>   project-forget-projects-under
>   project-forget-zombie-projects

It is also my impression that the tabulated list doesn't make too much
sense (for now).

> And whatever ambiguities about recursive search can be solved through
> better descriptions in docstrings.

Ok, I will update the patches and refine the documentation. My plan
would be to be non-recursive by default, and recurse if a prefix
argument is given.

> We could also add a hook like project-after-read-list-hook, which
> people would be able to use for cleanup, e.g. adding
> project-forget-zombie-projects to it. Not sure whether to add it there
> by default, though.

The question is does it make sense to not always forget zombies? This
depends on the real-world performance. If it is a noticeable burden (at
least on some systems), users might prefer adding the function to
kill-emacs-hook instead of a project-after-read-list-hook.

-- 
	Philip Kaludercic

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

On 22.09.2021 10:15, Philip Kaludercic wrote:

>> And whatever ambiguities about recursive search can be solved through
>> better descriptions in docstrings.
> 
> Ok, I will update the patches and refine the documentation. My plan
> would be to be non-recursive by default, and recurse if a prefix
> argument is given.

If you like. No recursive option seems also okay.

>> We could also add a hook like project-after-read-list-hook, which
>> people would be able to use for cleanup, e.g. adding
>> project-forget-zombie-projects to it. Not sure whether to add it there
>> by default, though.
> 
> The question is does it make sense to not always forget zombies? This
> depends on the real-world performance.

FWIW, speaking of Tramp, we can either avoid cleaning remote dirs, or at 
least make sure the connection is on before checking (and skip those 
where it is not).

And another way to limit the performance impact is to cap the number of 
projects in history.

> If it is a noticeable burden (at
> least on some systems), users might prefer adding the function to
> kill-emacs-hook instead of a project-after-read-list-hook.

kill-emacs-hook is also a good place to use it indeed. Maybe we don't 
need the other hook.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Juri Linkov]

> While I'm not necessarily a fan of the tabulated list approach, the above
> list looks sensible.

We already have a lot of 'list-*' commands for everything like
'list-packages', 'list-timers', etc. So sooner or later
'list-projects' will need to be added anyway, with the minimal
requirement of using that list for quicker and more visual
project switching with 'RET' like in 'list-buffers', and
also for issuing commands on a group of selected projects, etc.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

Juri Linkov <juri@linkov.net> writes:

>> While I'm not necessarily a fan of the tabulated list approach, the above
>> list looks sensible.
>
> We already have a lot of 'list-*' commands for everything like
> 'list-packages', 'list-timers', etc. So sooner or later
> 'list-projects' will need to be added anyway, with the minimal
> requirement of using that list for quicker and more visual
> project switching with 'RET' like in 'list-buffers', and
> also for issuing commands on a group of selected projects, etc.

I think it would make sense to have some kind of a way to name projects,
either by extracting the information from the VCS or via an
alias. When I tried implementing something like this, the only two
columns I could come up with to present were type (that isn't to
interesting to begin with) and project root. This seems a too meagre to
warren a list-projects function.

-- 
	Philip Kaludercic

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

On 22.09.2021 19:44, Philip Kaludercic wrote:
> I think it would make sense to have some kind of a way to name projects,
> either by extracting the information from the VCS or via an
> alias.

If we add this feature, I think most of the names will be more or less 
equal to project root's base name. So it's not like this will add much 
in the way of variety to the table.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 22.09.2021 19:44, Philip Kaludercic wrote:
>> I think it would make sense to have some kind of a way to name projects,
>> either by extracting the information from the VCS or via an
>> alias.
>
> If we add this feature, I think most of the names will be more or less
> equal to project root's base name. So it's not like this will add much
> in the way of variety to the table.

Currently yes. But if we were to store more metadata about a project
(nickname, first access, last access, etc.) maybe it might make more
sense.

-- 
	Philip Kaludercic

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

On 22.09.2021 21:06, Philip Kaludercic wrote:
> Currently yes. But if we were to store more metadata about a project
> (nickname, first access, last access, etc.) maybe it might make more
> sense.

That's possible, yes.

We could start storing access times right now, if you like. I don't know 
where else we would use them, though.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 22.09.2021 21:06, Philip Kaludercic wrote:
>> Currently yes. But if we were to store more metadata about a project
>> (nickname, first access, last access, etc.) maybe it might make more
>> sense.
>
> That's possible, yes.
>
> We could start storing access times right now, if you like. I don't
> know where else we would use them, though.

I don't think there is any use for that either right now, so I wouldn't
argue that it should be added immediately.

-- 
	Philip Kaludercic

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 72 bytes --]


Here is the updated patch, with the new names and optional recursion:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-aggregate-project-discovery-and-maintenance-func.patch --]
[-- Type: text/x-diff, Size: 4642 bytes --]

From 4779aae9d1c18cd2dc2b8f54322b48b0e11ac5fb Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Tue, 31 Aug 2021 14:12:13 +0200
Subject: [PATCH] Add aggregate project discovery and maintenance functions

* project.el (project-remember-project): Add optional no-write argument
(project-remember-projects-under): Add command
(project-forget-zombie-projects): Add command
(project-forget-known-projects): Add command
---
 lisp/progmodes/project.el | 72 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 69 insertions(+), 3 deletions(-)

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index ebd21d4b60..977b1ae185 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1290,9 +1290,10 @@ project--write-project-list
       (write-region nil nil filename nil 'silent))))
 
 ;;;###autoload
-(defun project-remember-project (pr)
+(defun project-remember-project (pr &optional no-write)
   "Add project PR to the front of the project list.
-Save the result in `project-list-file' if the list of projects has changed."
+Save the result in `project-list-file' if the list of projects
+has changed, and NO-WRITE is nil."
   (project--ensure-read-project-list)
   (let ((dir (project-root pr)))
     (unless (equal (caar project--list) dir)
@@ -1300,7 +1301,8 @@ project-remember-project
         (when (equal dir (car ent))
           (setq project--list (delq ent project--list))))
       (push (list dir) project--list)
-      (project--write-project-list))))
+      (unless no-write
+        (project--write-project-list)))))
 
 (defun project--remove-from-project-list (project-root report-message)
   "Remove directory PROJECT-ROOT of a missing project from the project list.
@@ -1357,6 +1359,70 @@ project-execute-extended-command
   (let ((default-directory (project-root (project-current t))))
     (call-interactively #'execute-extended-command)))
 
+(defun project-remember-projects-under (dir &optional recursive)
+  "Index all projects below a directory DIR.
+If RECURSIVE is non-nil, recurse into all subdirectories to find
+more projects.  After finishing, a message is printed summarizing
+the progress.  The function returns the number of detected
+projects."
+  (interactive "DDirectory: \nP")
+  (project--ensure-read-project-list)
+  (let ((queue (directory-files dir t nil t)) (count 0)
+        (known (make-hash-table
+                :size (* 2 (length project--list))
+                :test #'equal )))
+    (dolist (project (mapcar #'car project--list))
+      (puthash project t known))
+    (while queue
+      (when-let ((subdir (pop queue))
+                 ((file-directory-p subdir))
+                 ((not (gethash subdir known))))
+        (when-let (pr (project--find-in-directory subdir))
+          (project-remember-project pr t)
+          (message "Found %s..." (project-root pr))
+          (setq count (1+ count)))
+        (when (and recursive (file-symlink-p subdir))
+          (setq queue (nconc (directory-files subdir t nil t) queue))
+          (puthash subdir t known))))
+    (unless (eq recursive 'in-progress)
+      (if (zerop count)
+          (message "No projects were found")
+        (project--write-project-list)
+        (message "%d project%s were found"
+                 count (if (= count 1) "" "s"))))
+    count))
+
+(defun project-forget-zombie-projects ()
+  "Forget all known projects that don't exist any more."
+  (interactive)
+  (dolist (proj (project-known-project-roots))
+    (unless (file-exists-p proj)
+      (project-remove-known-project proj))))
+
+(defun project-forget-known-projects (dir &optional recursive)
+  "Forget all known projects below a directory DIR.
+If RECURSIVE is non-nil, recurse into all subdirectories to
+remove all known projects.  After finishing, a message is printed
+summarizing the progress.  The function returns the number of
+forgotten projects."
+  (interactive "DDirectory: \nP")
+  (let ((count 0))
+    (if recursive
+        (dolist (proj (project-known-project-roots))
+          (when (file-in-directory-p proj dir)
+            (project-remove-known-project proj)
+            (setq count (1+ count))))
+      (dolist (proj (project-known-project-roots))
+        (when (file-equal-p (file-name-directory proj) dir)
+          (project-remove-known-project proj)
+          (setq count (1+ count)))))
+    (if (zerop count)
+        (message "No projects were forgotten")
+      (project--write-project-list)
+      (message "%d project%s were forgotten"
+               count (if (= count 1) "" "s")))
+    count))
+
 \f
 ;;; Project switching
 
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 24 bytes --]


-- 
	Philip Kaludercic

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

On 22.09.2021 21:55, Philip Kaludercic wrote:
> +      (project-remove-known-project proj))))

This one is called 'project-forget-project' now.

> +(defun project-forget-known-projects (dir &optional recursive)

And let's call this one 'project-forget-projects-under'.

Looks good otherwise, thanks.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 22.09.2021 21:55, Philip Kaludercic wrote:
>> +      (project-remove-known-project proj))))
>
> This one is called 'project-forget-project' now.
>
>> +(defun project-forget-known-projects (dir &optional recursive)
>
> And let's call this one 'project-forget-projects-under'.
>
> Looks good otherwise, thanks.

Sorry about that, forgot to byte-compile before preparing the
patch. This should fix the issues:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-aggregate-project-discovery-and-maintenance-func.patch --]
[-- Type: text/x-diff, Size: 4624 bytes --]

From 45492ec9fb9f2706f6ac0e7317ec1b0f7ce25091 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Tue, 31 Aug 2021 14:12:13 +0200
Subject: [PATCH] Add aggregate project discovery and maintenance functions

* project.el (project-remember-project): Add optional no-write argument
(project-remember-projects-under): Add command
(project-forget-zombie-projects): Add command
(project-forget-projects-under): Add command
---
 lisp/progmodes/project.el | 72 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 69 insertions(+), 3 deletions(-)

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 9b63f4b1bc..57a961c260 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1296,9 +1296,10 @@ project--write-project-list
       (write-region nil nil filename nil 'silent))))
 
 ;;;###autoload
-(defun project-remember-project (pr)
+(defun project-remember-project (pr &optional no-write)
   "Add project PR to the front of the project list.
-Save the result in `project-list-file' if the list of projects has changed."
+Save the result in `project-list-file' if the list of projects
+has changed, and NO-WRITE is nil."
   (project--ensure-read-project-list)
   (let ((dir (project-root pr)))
     (unless (equal (caar project--list) dir)
@@ -1306,7 +1307,8 @@ project-remember-project
         (when (equal dir (car ent))
           (setq project--list (delq ent project--list))))
       (push (list dir) project--list)
-      (project--write-project-list))))
+      (unless no-write
+        (project--write-project-list)))))
 
 (defun project--remove-from-project-list (project-root report-message)
   "Remove directory PROJECT-ROOT of a missing project from the project list.
@@ -1363,6 +1365,70 @@ project-execute-extended-command
   (let ((default-directory (project-root (project-current t))))
     (call-interactively #'execute-extended-command)))
 
+(defun project-remember-projects-under (dir &optional recursive)
+  "Index all projects below a directory DIR.
+If RECURSIVE is non-nil, recurse into all subdirectories to find
+more projects.  After finishing, a message is printed summarizing
+the progress.  The function returns the number of detected
+projects."
+  (interactive "DDirectory: \nP")
+  (project--ensure-read-project-list)
+  (let ((queue (directory-files dir t nil t)) (count 0)
+        (known (make-hash-table
+                :size (* 2 (length project--list))
+                :test #'equal )))
+    (dolist (project (mapcar #'car project--list))
+      (puthash project t known))
+    (while queue
+      (when-let ((subdir (pop queue))
+                 ((file-directory-p subdir))
+                 ((not (gethash subdir known))))
+        (when-let (pr (project--find-in-directory subdir))
+          (project-remember-project pr t)
+          (message "Found %s..." (project-root pr))
+          (setq count (1+ count)))
+        (when (and recursive (file-symlink-p subdir))
+          (setq queue (nconc (directory-files subdir t nil t) queue))
+          (puthash subdir t known))))
+    (unless (eq recursive 'in-progress)
+      (if (zerop count)
+          (message "No projects were found")
+        (project--write-project-list)
+        (message "%d project%s were found"
+                 count (if (= count 1) "" "s"))))
+    count))
+
+(defun project-forget-zombie-projects ()
+  "Forget all known projects that don't exist any more."
+  (interactive)
+  (dolist (proj (project-known-project-roots))
+    (unless (file-exists-p proj)
+      (project-forget-project proj))))
+
+(defun project-forget-projects-under (dir &optional recursive)
+  "Forget all known projects below a directory DIR.
+If RECURSIVE is non-nil, recurse into all subdirectories to
+remove all known projects.  After finishing, a message is printed
+summarizing the progress.  The function returns the number of
+forgotten projects."
+  (interactive "DDirectory: \nP")
+  (let ((count 0))
+    (if recursive
+        (dolist (proj (project-known-project-roots))
+          (when (file-in-directory-p proj dir)
+            (project-forget-project proj)
+            (setq count (1+ count))))
+      (dolist (proj (project-known-project-roots))
+        (when (file-equal-p (file-name-directory proj) dir)
+          (project-forget-project proj)
+          (setq count (1+ count)))))
+    (if (zerop count)
+        (message "No projects were forgotten")
+      (project--write-project-list)
+      (message "%d project%s were forgotten"
+               count (if (= count 1) "" "s")))
+    count))
+
 \f
 ;;; Project switching
 
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 24 bytes --]


-- 
	Philip Kaludercic

bug#50297: 28.0.50; Aggregate project functions for project.el

[Dmitry Gutov]

On 23.09.2021 13:46, Philip Kaludercic wrote:
> Sorry about that, forgot to byte-compile before preparing the
> patch. This should fix the issues:

Thanks. You can go ahead and install.

Just add NEWS entries for the commands.

bug#50297: 28.0.50; Aggregate project functions for project.el

[Philip Kaludercic]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 23.09.2021 13:46, Philip Kaludercic wrote:
>> Sorry about that, forgot to byte-compile before preparing the
>> patch. This should fix the issues:
>
> Thanks. You can go ahead and install.
>
> Just add NEWS entries for the commands.

Done and pushed.

-- 
	Philip Kaludercic