bug#67687: Feature request: automatic tags management

[Stefan Kangas <stefankangas@gmail.com>]

My review below.  I wasn't paying attention to the full discussion, so
please just ignore any points that you have already discussed with Eli.

Dmitry Gutov <dmitry@gutov.dev> writes:

> diff --git a/etc/NEWS b/etc/NEWS
> index f82564946b7..6d6bca187de 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -1243,6 +1243,11 @@ the needs of users with red-green or blue-yellow color deficiency.
>  The Info manual "(modus-themes) Top" describes the details and
>  showcases all their customization options.
>
> +** New global minor mode 'etags-regen-mode'.
> +This minor mode generates the tags table automatically based on the
> +current project configuration, and later updates it as you edit the
> +files and save the changes.

Consider referring to the relevant section in the info manual.

> +
>  \f
>  * Incompatible Lisp Changes in Emacs 30.1
>
> diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
> new file mode 100644
> index 00000000000..e1fca1c4e44
> --- /dev/null
> +++ b/lisp/progmodes/etags-regen.el
> @@ -0,0 +1,424 @@
> +;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
> +
> +;; Copyright (C) 2021-2023 Free Software Foundation, Inc.
> +
> +;; Author: Dmitry Gutov <dmitry@gutov.dev>
> +;; Keywords: tools
> +
> +;; This file is part of GNU Emacs.
> +
> +;; GNU Emacs is free software: you can redistribute it and/or modify
> +;; it under the terms of the GNU General Public License as published by
> +;; the Free Software Foundation, either version 3 of the License, or
> +;; (at your option) any later version.
> +
> +;; GNU Emacs is distributed in the hope that it will be useful,
> +;; but WITHOUT ANY WARRANTY; without even the implied warranty of
> +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;; GNU General Public License for more details.
> +
> +;; You should have received a copy of the GNU General Public License
> +;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
> +
> +;;; Commentary:
> +
> +;; Simple automatic tags generation with updates on save.
> +;;
> +;; This mode provides automatic indexing for Emacs "go to definition"
> +;; feature, the `xref-go-forward' command (bound to `M-.' by default).
> +;;
> +;; At the moment reindexing works off before/after-save-hook, but to
> +;; handle more complex changes (for example, the user switching to
> +;; another branch from the terminal) we can look into plugging into
> +;; something like `filenotify'.
> +;;
> +;; Note that this feature disables itself if the user has some tags
> +;; table already visited (with `M-x visit-tags-table', or through an
> +;; explicit prompt triggered by some feature that requires tags).
> +
> +;;; Code:
> +
> +(require 'cl-lib)
> +
> +(defgroup etags-regen nil
> +  "Auto-(re)generating tags."
> +  :group 'tools)

How about:

    "Auto-generate \"tags\" file."

> +
> +(defvar etags-regen--tags-file nil)
> +(defvar etags-regen--tags-root nil)
> +(defvar etags-regen--new-file nil)
> +
> +(declare-function project-root "project")
> +(declare-function project-files "project")
> +(declare-function dired-glob-regexp "dired")
> +
> +(defcustom etags-regen-program (executable-find "etags")
> +  "Name of the etags program used by `etags-regen-mode'.
> +
> +If you only have `ctags' installed, you can also set this to
> +\"ctags -e\".  Some features might not be supported this way."
> +  ;; Always having our 'etags' here would be easier, but we can't
> +  ;; always rely on it being installed.  So it might be ctags's etags.
> +  :type 'file
> +  :version "30.1")
> +
> +(defcustom etags-regen-tags-file "TAGS"
> +  "Name of the tags file to create inside the project by `etags-regen-mode'.
> +
> +The value should either be a simple file name (no directory
> +specified), or a function that accepts the project root directory
> +and returns a distinct absolute file name for its tags file.  The
> +latter possibility is useful when you prefer to store the tag
> +files somewhere else, for example in `temporary-file-directory'."
> +  :type '(choice (string :tag "File name")
> +                 (function :tag "Function that returns file name"))
> +  :version "30.1")

Did you consider making it always store the TAGS files somewhere in
`temporary-file-directory'?  They should be rather ephemereal in any
case, I think?

In that case, we could perhaps even ignore an existing TAGS file, if
this mode is enabled.  Perhaps as an option.

> +
> +(defcustom etags-regen-program-options nil
> +  "List of additional options for etags program invoked by `etags-regen-mode'."
> +  :type '(repeat string)
> +  :version "30.1")

Perhaps add:

    See the full list of options that `etags' accepts in Info node
    `(emacs) Create Tags Table'.

Should this be marked unsafe?  Actually, same question for all of the
above defcustoms, given that we use `shell-command-to-string'.

Speaking of which, should we have more `shell-quote-argument' below?
I didn't look at every example, but maybe you did.

> +
> +(defcustom etags-regen-regexp-alist nil
> +  "Mapping of languages to etags regexps for `etags-regen-mode'.
> +
> +These regexps are used in addition to the tags made with the
> +standard parsing based on the language.
> +
> +The value must be a list of conses (LANGUAGES . TAG-REGEXPS)
> +where both car and cdr are lists of strings.

I think that should better be:

    where both LANGUAGES and TAG-REGEXPS are lists of strings.

I'm not sure we should say "conses" there, or should we?  I think we
usually prefer something like:

    The value is a list where each element has the form
    (LANGUAGES . TAG-REGEXPS)

I think this is better because it sounds less foreign to random users
with no background in ELisp.

But Eli is much better than me at this.  :-)

> +
> +Each language should be one of the recognized by etags, see
> +`etags --help'.  Each tag regexp should be a string in the format
> +as documented for the `--regex' arguments (without `{language}').
   ^^

Nit, but I think "as" could be scratched.

> +
> +We currently support only Emacs's etags program with this option."
> +  :type '(repeat
> +          (cons
> +           :tag "Languages group"
> +           (repeat (string :tag "Language name"))
> +           (repeat (string :tag "Tag Regexp"))))
> +  :version "30.1")
> +
> +;;;###autoload
> +(put 'etags-regen-regexp-alist 'safe-local-variable
> +     (lambda (value)
> +       (and (listp value)
> +            (seq-every-p
> +             (lambda (group)
> +               (and (consp group)
> +                    (listp (car group))
> +                    (listp (cdr group))
> +                    (seq-every-p #'stringp (car group))
> +                    (seq-every-p #'stringp (cdr group))))
> +             value))))
> +
> +;; We have to list all extensions: etags falls back to Fortran
> +;; when it cannot determine the type of the file.

(A battle-tested default, if nothing else.  ;-)

> +;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
> +(defcustom etags-regen-file-extensions
> +  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
> +    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
> +    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
> +  "Code file extensions for `etags-regen-mode'.
> +File extensions to generate the tags for."
> +  :type '(repeat (string :tag "File extension"))
> +  :version "30.1")
> +
> +;;;###autoload
> +(put 'etags-regen-file-extensions 'safe-local-variable
> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
> +
> +;; FIXME: We don't support root anchoring yet.

What is root anchoring?  Does it deserve a sentence that explains what
it is?

> +(defcustom etags-regen-ignores nil
> +  "Additional ignore rules, in the format of `project-ignores'."
> +  :type '(repeat
> +          (string :tag "Glob to ignore"))
> +  :version "30.1")
> +
> +;;;###autoload
> +(put 'etags-regen-ignores 'safe-local-variable
> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
> +
> +(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
> +
> +(defvar etags-regen--rescan-files-limit 100)
> +
> +(defun etags-regen--all-mtimes (proj)
> +  (let ((files (etags-regen--all-files proj))
> +        (mtimes (make-hash-table :test 'equal))
> +        file-name-handler-alist)
> +    (dolist (f files)
> +      (condition-case nil
> +          (puthash f
> +                   (file-attribute-modification-time
> +                    (file-attributes f))
> +                   mtimes)
> +        (file-missing nil)))
> +    mtimes))

Could we use file notifications for this?  Maybe as a future
improvement.

Other than that, LGTM.

Thanks again for working on this.