bug#67687: Feature request: automatic tags management

bug#67687: Feature request: automatic tags management

[Jon Eskin]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]

Sublime text and the vim extension vim-gutentags (
https://github.com/ludovicchabant/vim-gutentags) allow you to open projects
and immediately go to symbol definitions and references without any
configuration or setup. They also handle background indexing as you work.

It would be very nice if Emacs had an option to work the same way, where I
could open a project and immediately navigate symbols built from the
project.el root, without having to manually futz around with tags.

[-- Attachment #2: Type: text/html, Size: 574 bytes --]

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

Hi!

On 07/12/2023 13:43, Jon Eskin wrote:
> Sublime text and the vim extension vim-gutentags 
> (https://github.com/ludovicchabant/vim-gutentags 
> <https://github.com/ludovicchabant/vim-gutentags>) allow you to open 
> projects and immediately go to symbol definitions and references without 
> any configuration or setup. They also handle background indexing as you 
> work.
> 
> It would be very nice if Emacs had an option to work the same way, where 
> I could open a project and immediately navigate symbols built from the 
> project.el root, without having to manually futz around with tags.

I have some related work that's been lying in a drawer as of late.

Do you know what gutentags does when a file is deleted, or added 
externally, or you switch to a different Git branch and many files 
change their contents at once?

bug#67687: Feature request: automatic tags management

[Jon Eskin]

Hi Dmitry!

> I have some related work that's been lying in a drawer as of late.
> 
> Do you know what gutentags does when a file is deleted, or added externally, or you switch to a different Git branch and many files change their contents at once?

To my knowledge, changes to tags due to files being added or deleted are regenerated only when vim is started inside the project. Switching to a different git branch should work the same, because it’s only looking at the filesystem.

If the editor is already running, such changes will not be picked up; even if you try to navigate to a tag that no longer exists, I do not believe that it will regenerate tags automatically until you close and re-open the editor.

When you save a file inside the editor, it will wipe out tags for that individual file and regenerate them.

Here are some docs for customization options you might find interesting:

  *gutentags_generate_on_missing*
g:gutentags_generate_on_missing
                        If set to 1, Gutentags will start generating an initial
                        tag file if a file is open in a project where no tags
                        file is found. See |gutentags_project_root| for how
                        Gutentags locates the project.
                        When set to 0, Gutentags will only generate the first
                        time the file is saved (if
                        |gutentags_generate_on_write| is set to 1), or when
                        |GutentagsUpdate| or |GutentagsGenerate| is run.
                        Defaults to 1.

                                                *gutentags_generate_on_new*
g:gutentags_generate_on_new
                        If set to 1, Gutentags will start generating the tag
                        file when a new project is open. A new project is
                        considered open when a buffer is created for a file
                        whose corresponding tag file has not been "seen" yet
                        in the current Vim session -- which pretty much means
                        when you open the first file in a given source control
                        repository.
                        When set to 0, Gutentags won't do anything special.
                        See also |gutentags_generate_on_missing| and
                        |gutentags_generate_on_write|.
                        Defaults to 1.

                                                *gutentags_generate_on_write*
g:gutentags_generate_on_write
                        If set to 1, Gutentags will update the current
                        project's tag file when a file inside that project is
                        saved. See |gutentags_project_root| for how Gutentags
                        locates the project.
                        When set to 0, Gutentags won't do anything on save.
                        This means that the project's tag file won't reflect
                        the latest changes, and you will have to run
                        |GutentagsUpdate| manually.
                        Defaults to 1.

                                            *gutentags_generate_on_empty_buffer*
g:gutentags_generate_on_empty_buffer
                        If set to 1, Gutentags will start generating the tag
                        file even if there's no buffer currently open, as long
                        as the current working directory (as returned by
                        |:cd|) is inside a known project.
                        This is useful if you want Gutentags to generate the
                        tag file right after opening Vim.
                        Defaults to 0.

More can be found here: https://github.com/ludovicchabant/vim-gutentags/blob/master/doc/gutentags.txt

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1692 bytes --]

Hi Jon!

On 07/12/2023 21:57, Jon Eskin wrote:

>> I have some related work that's been lying in a drawer as of late.
>>
>> Do you know what gutentags does when a file is deleted, or added externally, or you switch to a different Git branch and many files change their contents at once?
> To my knowledge, changes to tags due to files being added or deleted are regenerated only when vim is started inside the project. Switching to a different git branch should work the same, because it’s only looking at the filesystem.
> 
> If the editor is already running, such changes will not be picked up; even if you try to navigate to a tag that no longer exists, I do not believe that it will regenerate tags automatically until you close and re-open the editor.
> 
> When you save a file inside the editor, it will wipe out tags for that individual file and regenerate them.

All right, sounds like this kind of limited guarantees (tags getting out 
of date occasionally, e.g. when switching branches) is apparently okay.

See attached the aforementioned related work, with some updates and 
simplifications (e.g. this version of the patch doesn't require a change 
to project.el).

Usage: 'M-x etags-regen-mode' to turn it on (or customize this variable, 
to have it on from the beginning of each session), then as soon as you 
use features based on tags (such as M-. or completion) the table should 
get generated automatically for the current project, and then get 
updated when files are edited and saved.

Some features could be added later (such as asynchronous updates or -- 
someday -- filenotify based invalidations), but I think the current 
state is useful already.

Feedback welcome.

[-- Attachment #2: etags-regen.diff --]
[-- Type: text/x-patch, Size: 14311 bytes --]

diff --git a/.dir-locals.el b/.dir-locals.el
index e087aa89cd1..d308591c475 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -8,6 +8,12 @@
          (vc-git-annotate-switches . "-w")
          (bug-reference-url-format . "https://debbugs.gnu.org/%s")
 	 (diff-add-log-use-relative-names . t)
+         (etags-regen-lang-regexp-alist
+          .
+          ((("c" "objc") .
+            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
+             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
+         (etags-regen-ignores . ("test/manual/etags/"))
          (vc-prepare-patches-separately . nil)))
  (c-mode . ((c-file-style . "GNU")
             (c-noise-macro-names . ("INLINE" "NO_INLINE" "ATTRIBUTE_NO_SANITIZE_UNDEFINED"
diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
new file mode 100644
index 00000000000..ecb6e8ae2c3
--- /dev/null
+++ b/lisp/progmodes/etags-regen.el
@@ -0,0 +1,355 @@
+;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2021, 2023 Free Software Foundation, Inc.
+
+;; Author: Dmitry Gutov <dmitry@gutov.dev>
+;; Keywords: tools
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Simple tags generation with automatic invalidation.
+
+;;; Code:
+
+(require 'cl-lib)
+
+(defgroup etags-regen nil
+  "Auto-(re)generating tags."
+  :group 'tools)
+
+(defvar etags-regen--tags-file nil)
+(defvar etags-regen--tags-root nil)
+(defvar etags-regen--new-file nil)
+
+(declare-function project-root "project")
+(declare-function project-files "project")
+
+(defcustom etags-regen-program (executable-find "etags")
+  "Name of the etags executable."
+  ;; Always having our 'etags' here would be easier, but we can't
+  ;; always rely on it being installed.  So it might be ctags's etags.
+  :type 'file)
+
+(defcustom etags-regen-tags-file "TAGS"
+  "Name of the tags file to create inside the project."
+  :type 'string)
+
+(defcustom etags-regen-program-options nil
+  "List of additional options to pass to the etags program."
+  :type '(repeat string))
+
+(defcustom etags-regen-lang-regexp-alist nil
+  "Mapping of languages to additional regexps for tags.
+
+Each language should be one of the recognized by etags, see
+`etags --help'.  Each tag regexp should be a string in the format
+as documented for the `--regex' arguments.
+
+We support only Emacs's etags program with this option."
+  :type '(repeat
+          (cons
+           :tag "Languages group"
+           (repeat (string :tag "Language name"))
+           (repeat (string :tag "Tag Regexp")))))
+
+;;;###autoload
+(put 'etags-regen-lang-regexp-alist 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p
+             (lambda (group)
+               (and (consp group)
+                    (listp (car group))
+                    (listp (cdr group))
+                    (seq-every-p
+                     (lambda (lang)
+                       (and (stringp lang)
+                            (string-match-p "\\`[a-z*+]+\\'" lang)))
+                     (car group))
+                    (seq-every-p #'stringp (cdr group))))
+             value))))
+
+;; XXX: We have to list all extensions: etags falls back to Fortran.
+;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
+(defcustom etags-regen-file-extensions
+  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
+    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
+    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
+  "Code file extensions.
+
+File extensions to generate the tags for."
+  :type '(repeat (string :tag "File extension")))
+
+;;;###autoload
+(put 'etags-regen-file-extensions 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p
+             (lambda (ext)
+               (and (stringp ext)
+                    (string-match-p "\\`[a-zA-Z0-9]+\\'" ext)))
+             value))))
+
+;; FIXME: Only plain substrings supported currently.
+(defcustom etags-regen-ignores nil
+  "Additional ignore rules, in the format of `project-ignores'."
+  :type '(repeat
+          (string :tag "Glob to ignore")))
+
+;;;###autoload
+(put 'etags-regen-ignores 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p #'stringp value))))
+
+(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
+
+(defun etags-regen--all-mtimes (proj)
+  (let ((files (etags-regen--all-files proj))
+        (mtimes (make-hash-table :test 'equal))
+        file-name-handler-alist)
+    (dolist (f files)
+      (condition-case nil
+          (puthash f
+                   (file-attribute-modification-time
+                    (file-attributes f))
+                   mtimes)
+        (file-missing nil)))
+    mtimes))
+
+(defun etags-regen--refresh (proj)
+  (save-excursion
+    (let* ((tags-file (expand-file-name etags-regen-tags-file
+                                        (project-root proj)))
+           (tags-mtime (file-attribute-modification-time
+                        (file-attributes tags-file)))
+           (all-mtimes (etags-regen--all-mtimes proj))
+           added-files
+           changed-files
+           removed-files)
+      (etags-regen--visit-table tags-file (project-root proj))
+      (set-buffer (get-file-buffer tags-file))
+      (dolist (file (tags-table-files))
+        (let ((mtime (gethash file all-mtimes)))
+          (cond
+           ((null mtime)
+            (push file removed-files))
+           ((time-less-p tags-mtime mtime)
+            (push file changed-files)
+            (remhash file all-mtimes))
+           (t
+            (remhash file all-mtimes)))))
+      (maphash
+       (lambda (key _value)
+         (push key added-files))
+       all-mtimes)
+      (if (> (+ (length added-files)
+                (length changed-files)
+                (length removed-files))
+             100)
+          (progn
+            (message "etags-regen: Too many changes, falling back to full rescan")
+            (etags-regen--tags-cleanup))
+        (dolist (file (nconc removed-files changed-files))
+          (etags-regen--remove-tag file))
+        (when (or changed-files added-files)
+          (apply #'etags-regen--append-tags
+                 (nconc changed-files added-files)))
+        (when (or changed-files added-files removed-files)
+          (let ((save-silently t)
+                (message-log-max nil))
+            (save-buffer 0)))))))
+
+(defun etags-regen--maybe-generate ()
+  (let ((proj))
+    (when (and etags-regen--tags-root
+               (not (file-in-directory-p default-directory
+                                         etags-regen--tags-root)))
+      (etags-regen--tags-cleanup))
+    (when (and (not etags-regen--tags-root)
+               (file-exists-p (expand-file-name
+                               etags-regen-tags-file
+                               (project-root
+                                (setq proj (project-current))))))
+      (message "Found existing tags table, refreshing...")
+      (etags-regen--refresh proj))
+    (when (and (not (or tags-file-name
+                        tags-table-list))
+               (setq proj (or proj (project-current))))
+      (message "Generating new tags table...")
+      (let ((start (time-to-seconds)))
+        (etags-regen--tags-generate proj)
+        (message "...done (%.2f s)" (- (time-to-seconds) start))))))
+
+(defun etags-regen--all-files (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         ;; TODO: Make the scanning more efficient, e.g. move the
+         ;; filtering by glob to project (project-files-filtered...).
+         (files (project-files proj))
+         (match-re (concat
+                    "\\."
+                    (regexp-opt etags-regen-file-extensions)
+                    "\\'")))
+    (cl-delete-if
+     (lambda (f) (or (not (string-match-p match-re f))
+                ;; FIXME: Handle etags-regen-ignores properly.
+                (string-match-p "/\\.#" f)
+                (cl-some (lambda (ignore) (string-search ignore f))
+                         etags-regen-ignores)))
+     files)))
+
+(defun etags-regen--tags-generate (proj)
+  (require 'dired)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         (files (etags-regen--all-files proj))
+         (tags-file (expand-file-name etags-regen-tags-file root))
+         (ctags-p (etags-regen--ctags-p))
+         (command (format "%s %s %s - -o %s"
+                          etags-regen-program
+                          (mapconcat #'identity
+                                     (etags-regen--build-program-options ctags-p)
+                                     " ")
+                          ;; ctags's etags requires '-L' for stdin input.
+                          (if ctags-p "-L" "")
+                          tags-file)))
+    (with-temp-buffer
+      (mapc (lambda (f)
+              (insert f "\n"))
+            files)
+      (shell-command-on-region (point-min) (point-max) command
+                               nil nil etags-regen--errors-buffer-name t))
+    (etags-regen--visit-table tags-file root)))
+
+(defun etags-regen--visit-table (tags-file root)
+  ;; Invalidate the scanned tags after any change is written to disk.
+  (add-hook 'after-save-hook #'etags-regen--update-file)
+  (add-hook 'before-save-hook #'etags-regen--mark-as-new)
+  (setq etags-regen--tags-file tags-file
+        etags-regen--tags-root root)
+  (visit-tags-table etags-regen--tags-file))
+
+(defun etags-regen--ctags-p ()
+  (string-search "Ctags"
+                 (shell-command-to-string
+                  (format "%s --version" etags-regen-program))))
+
+(defun etags-regen--build-program-options (ctags-p)
+  (when (and etags-regen-lang-regexp-alist ctags-p)
+    (user-error "etags-regen-lang-regexp-alist is not supported with Ctags"))
+  (nconc
+   (mapcan
+    (lambda (group)
+      (mapcan
+       (lambda (lang)
+         (mapcar (lambda (regexp)
+                   (concat "--regex="
+                           (shell-quote-argument
+                            (format "{%s}%s" lang regexp))))
+                 (cdr group)))
+       (car group)))
+    etags-regen-lang-regexp-alist)
+   etags-regen-program-options))
+
+(defun etags-regen--update-file ()
+  ;; TODO: Maybe only do this when Emacs is idle for a bit.  Or defer
+  ;; the updates and do them later in bursts when the table is used.
+  (let ((file-name buffer-file-name)
+        (tags-file-buf (get-file-buffer etags-regen--tags-file))
+        pr should-scan)
+    (save-excursion
+      (when tags-file-buf
+        (cond
+         ((and etags-regen--new-file
+               (kill-local-variable 'etags-regen--new-file)
+               (setq pr (project-current))
+               (equal (project-root pr) etags-regen--tags-root)
+               (member file-name (project-files pr)))
+          (set-buffer tags-file-buf)
+          (setq should-scan t))
+         ((progn (set-buffer tags-file-buf)
+                 (etags-regen--remove-tag file-name))
+          (setq should-scan t))))
+      (when should-scan
+        (etags-regen--append-tags file-name)
+        (let ((save-silently t)
+              (message-log-max nil))
+          (save-buffer 0))))))
+
+(defun etags-regen--remove-tag (file-name)
+  (goto-char (point-min))
+  (when (search-forward (format "\f\n%s," file-name) nil t)
+    (let ((start (match-beginning 0)))
+      (search-forward "\f\n" nil 'move)
+      (let ((inhibit-read-only t))
+        (delete-region start
+                       (if (eobp)
+                           (point)
+                         (- (point) 2)))))
+    t))
+
+(defun etags-regen--append-tags (&rest file-names)
+  (goto-char (point-max))
+  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
+        (inhibit-read-only t))
+    ;; FIXME: call-process is significantly faster, though.
+    ;; Like 10ms vs 20ms here.
+    (shell-command
+     (format "%s %s %s -o -"
+             etags-regen-program (mapconcat #'identity options " ")
+             (mapconcat #'identity file-names " "))
+     t etags-regen--errors-buffer-name))
+  ;; FIXME: Is there a better way to do this?
+  ;; Completion table is the only remaining place where the
+  ;; update is not incremental.
+  (setq-default tags-completion-table nil))
+
+(defun etags-regen--mark-as-new ()
+  (unless buffer-file-number
+    (setq-local etags-regen--new-file t)))
+
+(defun etags-regen--tags-cleanup ()
+  (when etags-regen--tags-file
+    (let ((buffer (get-file-buffer etags-regen--tags-file)))
+      (and buffer
+           (kill-buffer buffer)))
+    (setq tags-file-name nil
+          tags-table-list nil
+          etags-regen--tags-file nil
+          etags-regen--tags-root nil))
+  (remove-hook 'after-save-hook #'etags-regen--update-file)
+  (remove-hook 'before-save-hook #'etags-regen--mark-as-new))
+
+;;;###autoload
+(define-minor-mode etags-regen-mode
+  "Generate tags automatically."
+  :global t
+  (if etags-regen-mode
+      (progn
+        (advice-add 'etags--xref-backend :before
+                    #'etags-regen--maybe-generate)
+        (advice-add 'tags-completion-at-point-function :before
+                    #'etags-regen--maybe-generate))
+    (advice-remove 'etags--xref-backend #'etags-regen--maybe-generate)
+    (advice-remove 'tags-completion-at-point-function #'etags-regen--maybe-generate)
+    (etags-regen--tags-cleanup)))
+
+(provide 'etags-regen)
+
+;;; etags-regen.el ends here

bug#67687: Feature request: automatic tags management

[Jon Eskin]

> All right, sounds like this kind of limited guarantees (tags getting out of date occasionally, e.g. when switching branches) is apparently okay.
> 
> See attached the aforementioned related work, with some updates and simplifications (e.g. this version of the patch doesn't require a change to project.el).
> 
> Usage: 'M-x etags-regen-mode' to turn it on (or customize this variable, to have it on from the beginning of each session), then as soon as you use features based on tags (such as M-. or completion) the table should get generated automatically for the current project, and then get updated when files are edited and saved.
> 
> Some features could be added later (such as asynchronous updates or -- someday -- filenotify based invalidations), but I think the current state is useful already.
> 
> Feedback welcome.<etags-regen.diff>


Wow! This is excellent, worked like a charm on my end.

I’m going to get this set up on a few different machines and take some time to build feedback.

Thank you!

bug#67687: Feature request: automatic tags management

[Jon Eskin]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]


> Wow! This is excellent, worked like a charm on my end.
> 
> I’m going to get this set up on a few different machines and take some time to build feedback.
> 
> Thank you!

Just a quick follow up - I’ve been using this every day. I’ve been able to work without thinking about tags at all, and it’s been a huge QOL upgrade for me. 

The only road-bumps I had in the last few weeks were the following:

1) In some repositories I unknowingly had an existing universal ctags file “tags”. When I did, trying to goto-def would give an error like "user-error: File /Users/jon/development/c/clp/TAGS is not a valid tags table”. This was misleading, because the actual file on my filesystem is "/Users/jon/development/c/clp/tags”. Usually I differentiate ctags format from etags by the capitalization of the filename, and that error message makes it look like it’s an etags file. I’m guessing this has something to do with MacOS file system case insensitivity on the file system.

2) I had always generated tags with a path argument for some reason, so when I customized “Etags Regen Program” to “ctags -e -R .”, I got an error since it appends an -o option to the end. I just needed to glance at the source and experiment for a minute to realize I just needed to remove the “.”.

I don’t mean to suggest those are problems to be fixed, I can’t really think of any addition or change that would make sense to try to address them, but I figured I would report them in case you had any ideas.

Overall this package gets a big thumbs up from me and I think it would be a great inclusion in a future release. Thank you for sharing it!

Jon

[-- Attachment #2: Type: text/html, Size: 2176 bytes --]

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 20/12/2023 23:11, Jon Eskin wrote:
> 
>> Wow! This is excellent, worked like a charm on my end.
>>
>> I’m going to get this set up on a few different machines and take some 
>> time to build feedback.
>>
>> Thank you!
> 
> Just a quick follow up - I’ve been using this every day. I’ve been able 
> to work without thinking about tags at all, and it’s been a huge QOL 
> upgrade for me.

Excellent.

> The only road-bumps I had in the last few weeks were the following:
> 
> 1) In some repositories I unknowingly had an existing universal ctags 
> file “tags”. When I did, trying to goto-def would give an error like 
> "user-error: File /Users/jon/development/c/clp/TAGS is not a valid tags 
> table”. This was misleading, because the actual file on my filesystem is 
> "/Users/jon/development/c/clp/tags”. Usually I differentiate ctags 
> format from etags by the capitalization of the filename, and that error 
> message makes it look like it’s an etags file. I’m guessing this has 
> something to do with MacOS file system case insensitivity on the file 
> system.

Hm, the first advice I would give is customize 'etags-regen-tags-file' 
to something else -- e.g. "RETAGS"? Then it won't match existing files.

Later we could also add some file format verification (a regexp search 
or two), but probably not in the first version.

> 2) I had always generated tags with a path argument for some reason, so 
> when I customized “Etags Regen Program” to “ctags -e -R .”, I got an 
> error since it appends an -o option to the end. I just needed to glance 
> at the source and experiment for a minute to realize I just needed to 
> remove the “.”.

And "-R", I hope. Though it probably makes no difference, since none of 
the arguments passed to it are directories.

Yeah, "-R ." is antithetical to how the program is used. We can extend 
the :type argument for this option, adding "ctags -e" at least. Maybe a 
longer docstring as well.

> I don’t mean to suggest those are problems to be fixed, I can’t really 
> think of any addition or change that would make sense to try to address 
> them, but I figured I would report them in case you had any ideas.
> 
> Overall this package gets a big thumbs up from me and I think it would 
> be a great inclusion in a future release. Thank you for sharing it!

Thanks for testing!

That is a good sign (with another positive bit of feedback on Reddit 
yesterday), so I think it's time to ask the head maintainers what they 
think about the inclusion of this feature in the core now.

Eli/Stefan?

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Thu, 21 Dec 2023 02:24:01 +0200
> Cc: 67687@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> That is a good sign (with another positive bit of feedback on Reddit 
> yesterday), so I think it's time to ask the head maintainers what they 
> think about the inclusion of this feature in the core now.
> 
> Eli/Stefan?

I didn't hear about any disadvantages; are there any issues we need to
consider?

Does the etags test suite still pass after these changes?

Some comments based on a superficial look at the branch:

  . there are no updates for NEWS and the Emacs manual
  . the doc string of etags-regen-mode should explain more about what
    it does
  . the new -L switch to etags is not mentioned in --help and in the
    man page of etags
  . defcustoms don't have a :version tag
  . etags-regen-lang-regexp-alist could have a shorter name:
    etags-regen-regexp-alist, and its doc string should describe the
    form of the alist
  . in the safe-local-variable form of etags-regen-lang-regexp-alist,
    why do we force the language name to match a certain regexp, and
    likewise with extensions in etags-regen-file-extensions?
  . the shell command in etags-regen--all-mtimes is non-portable: it
    needs xargs and stat commands; please use
    directory-files-recursively with file-attributes instead, at least
    as fallback
  . I see several FIXMEs and TODOs in the code
  . I wonder whether we should make sure etags supports the new -L
    switch, and signal an error if not -- since you invoke etags
    via PATH, it is quite possible to invoke an older version, the one
    installed on the system, and not the one in the repository, as
    long as Emacs 30 is not installed
  . why does etags-regen--tags-generate require 'dired?
  . why do we have to use advice-add/remove in etags-regen-mode? can't
    we add hooks to the relevant functions instead?
  . you are removing a project-files method from project.el -- isn't
    that backward-incompatible?

Thanks.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 21/12/2023 09:40, Eli Zaretskii wrote:
>> Date: Thu, 21 Dec 2023 02:24:01 +0200
>> Cc: 67687@debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> That is a good sign (with another positive bit of feedback on Reddit
>> yesterday), so I think it's time to ask the head maintainers what they
>> think about the inclusion of this feature in the core now.
>>
>> Eli/Stefan?
> 
> I didn't hear about any disadvantages; are there any issues we need to
> consider?

Nothing major, I believe. But someone in charge should make a yes-or-no 
decision.

Also, the patch adds a new value to .dir-locals.el (to make 
etags-regen-mode easy to use for Emacs development right away).

If someone edits their development branch with an older (e.g. stable) 
version of Emacs, this will be one more unsafe-variable warning to 
handle at least once.

> Does the etags test suite still pass after these changes?

I don't see why it would be affected. The mode is off by default anyway 
-- though I think it might be a good idea to flip it on after some more 
testing. But that is up to you and Stefan K. as well.

> Some comments based on a superficial look at the branch:
> 
>    . there are no updates for NEWS and the Emacs manual

I'll certainly add something to NEWS. Not sure where and what should be 
in the manual.

>    . the doc string of etags-regen-mode should explain more about what
>      it does

Fair.

>    . the new -L switch to etags is not mentioned in --help and in the
>      man page of etags

You're probably looking at the Git branch in the repository. See instead 
the patch attached to this bug report. I've removed a bunch of less 
essential changes.

Some to be considered for later (the project.el changes, which make file 
listing somewhat faster), and some not. The -L flag is in the latter 
category.

>    . defcustoms don't have a :version tag

Will add.

>    . etags-regen-lang-regexp-alist could have a shorter name:
>      etags-regen-regexp-alist, and its doc string should describe the
>      form of the alist

It seemed that "lang" is good to have in the name, so the variable's 
meaning is more obvious, but I don't really mind changing it.

>    . in the safe-local-variable form of etags-regen-lang-regexp-alist,
>      why do we force the language name to match a certain regexp, and
>      likewise with extensions in etags-regen-file-extensions?

That's mostly defensive programming. Indeed, shell-quote-argument should 
deal with most of the potential security problems.

>    . the shell command in etags-regen--all-mtimes is non-portable: it
>      needs xargs and stat commands; please use
>      directory-files-recursively with file-attributes instead, at least
>      as fallback

This is already done in the latest version (etags-regen.diff in the bug 
attachments).

Which brings a problem: the mode is now likely unusable over Tramp in 
any project of significant size. Something to improve later.

>    . I see several FIXMEs and TODOs in the code

I don't plan on addressing any of those before the first checkin.

>    . I wonder whether we should make sure etags supports the new -L
>      switch, and signal an error if not -- since you invoke etags
>      via PATH, it is quite possible to invoke an older version, the one
>      installed on the system, and not the one in the repository, as
>      long as Emacs 30 is not installed

Yeah, this is not a problem anymore, see above.

>    . why does etags-regen--tags-generate require 'dired?

A remnant of some older code. Removed it.

>    . why do we have to use advice-add/remove in etags-regen-mode? can't
>      we add hooks to the relevant functions instead?

We can, and probably will do. What I like about the current solution, 
though, is that it can be published to GNU ELPA (as "core" package) 
verbatim, and work with older emacsen as well.

>    . you are removing a project-files method from project.el -- isn't
>      that backward-incompatible?

Not in the latest patch.

Thanks.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 160 bytes --]

On 21/12/2023 18:46, Dmitry Gutov wrote:
> See instead the patch attached to this bug report.

Here's an update, incorporating the feedback from here and there.

[-- Attachment #2: etags-regen-v2.diff --]
[-- Type: text/x-patch, Size: 16507 bytes --]

diff --git a/.dir-locals.el b/.dir-locals.el
index e087aa89cd1..d308591c475 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -8,6 +8,12 @@
          (vc-git-annotate-switches . "-w")
          (bug-reference-url-format . "https://debbugs.gnu.org/%s")
 	 (diff-add-log-use-relative-names . t)
+         (etags-regen-lang-regexp-alist
+          .
+          ((("c" "objc") .
+            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
+             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
+         (etags-regen-ignores . ("test/manual/etags/"))
          (vc-prepare-patches-separately . nil)))
  (c-mode . ((c-file-style . "GNU")
             (c-noise-macro-names . ("INLINE" "NO_INLINE" "ATTRIBUTE_NO_SANITIZE_UNDEFINED"
diff --git a/etc/NEWS b/etc/NEWS
index 90ff23b7937..3726655239e 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1208,6 +1208,11 @@ the needs of users with red-green or blue-yellow color deficiency.
 The Info manual "(modus-themes) Top" describes the details and
 showcases all their customization options.
 
+** New global minor mode 'etags-regen-mode'.
+This minor mode generates the tags table automatically based on the
+current project configuration, and later updates it as you edit the
+files and save the changes.
+
 \f
 * Incompatible Lisp Changes in Emacs 30.1
 
diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
new file mode 100644
index 00000000000..66c4178ae86
--- /dev/null
+++ b/lisp/progmodes/etags-regen.el
@@ -0,0 +1,384 @@
+;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2021, 2023 Free Software Foundation, Inc.
+
+;; Author: Dmitry Gutov <dmitry@gutov.dev>
+;; Keywords: tools
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Simple automatic tags generation with updates on save.
+;;
+;; The goal of this mode is to provide a feature that should be
+;; familiar to the users of certain lightweight programmer's editors,
+;; such as Sublime Text.  Which is "go to definition" with automatic
+;; indexing, added in ST3 (released in 2017).
+;;
+;; At the moment reindexing works off before/after-save-hook, but to
+;; handle more complex changes (e.g. the user switching to another
+;; branch from the terminal) we can look into plugging into something
+;; like `filenotify'.
+;;
+;; Note that this feature disables itself if the user has some tags
+;; table already visited (with `M-x visit-tags-table', or through an
+;; explicit prompt triggered by some feature that requires tags).
+
+;;; Code:
+
+(require 'cl-lib)
+
+(defgroup etags-regen nil
+  "Auto-(re)generating tags."
+  :group 'tools)
+
+(defvar etags-regen--tags-file nil)
+(defvar etags-regen--tags-root nil)
+(defvar etags-regen--new-file nil)
+
+(declare-function project-root "project")
+(declare-function project-files "project")
+
+(defcustom etags-regen-program (executable-find "etags")
+  "Name of the etags program.
+
+If you only have `ctags' installed, you can also set this to
+\"ctags -e\".  Some features might not be supported this way."
+  ;; Always having our 'etags' here would be easier, but we can't
+  ;; always rely on it being installed.  So it might be ctags's etags.
+  :type 'file)
+
+(defcustom etags-regen-tags-file "TAGS"
+  "Name of the tags file to create inside the project.
+
+This value should either be a simple file name (no directory
+specified), or a function that accepts a project root directory
+and returns a distinct file name for the tags file for it.  The
+latter option is most useful when you prefer to store the tag
+files somewhere outside -- e.g. in `temporary-file-directory'."
+  :type '(choice (string :tag "File name")
+                 (function :tag "Function that returns file name")))
+
+(defcustom etags-regen-program-options nil
+  "List of additional options to pass to the etags program."
+  :type '(repeat string))
+
+(defcustom etags-regen-regexp-alist nil
+  "Mapping of languages to additional regexps for tags.
+
+The value must be a list of conses (LANGUAGES . TAG-REGEXPS)
+where both car and cdr are lists of strings.
+
+Each language should be one of the recognized by etags, see
+`etags --help'.  Each tag regexp should be a string in the format
+as documented for the `--regex' arguments.
+
+We currently support only Emacs's etags program with this option."
+  :type '(repeat
+          (cons
+           :tag "Languages group"
+           (repeat (string :tag "Language name"))
+           (repeat (string :tag "Tag Regexp")))))
+
+;;;###autoload
+(put 'etags-regen-regexp-alist 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p
+             (lambda (group)
+               (and (consp group)
+                    (listp (car group))
+                    (listp (cdr group))
+                    (seq-every-p #'stringp (car group))
+                    (seq-every-p #'stringp (cdr group))))
+             value))))
+
+;; We have to list all extensions: etags falls back to Fortran
+;; when it cannot determine the type of the file.
+;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
+(defcustom etags-regen-file-extensions
+  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
+    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
+    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
+  "Code file extensions.
+
+File extensions to generate the tags for."
+  :type '(repeat (string :tag "File extension")))
+
+;;;###autoload
+(put 'etags-regen-file-extensions 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+;; FIXME: Only plain substrings supported currently.
+(defcustom etags-regen-ignores nil
+  "Additional ignore rules, in the format of `project-ignores'."
+  :type '(repeat
+          (string :tag "Glob to ignore")))
+
+;;;###autoload
+(put 'etags-regen-ignores 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
+
+(defun etags-regen--all-mtimes (proj)
+  (let ((files (etags-regen--all-files proj))
+        (mtimes (make-hash-table :test 'equal))
+        file-name-handler-alist)
+    (dolist (f files)
+      (condition-case nil
+          (puthash f
+                   (file-attribute-modification-time
+                    (file-attributes f))
+                   mtimes)
+        (file-missing nil)))
+    mtimes))
+
+(defun etags-regen--choose-tags-file (proj)
+  (if (functionp etags-regen-tags-file)
+      (funcall etags-regen-tags-file (project-root proj))
+    (expand-file-name etags-regen-tags-file (project-root proj))))
+
+(defun etags-regen--refresh (proj)
+  (save-excursion
+    (let* ((tags-file (etags-regen--choose-tags-file proj))
+           (tags-mtime (file-attribute-modification-time
+                        (file-attributes tags-file)))
+           (all-mtimes (etags-regen--all-mtimes proj))
+           added-files
+           changed-files
+           removed-files)
+      (etags-regen--visit-table tags-file (project-root proj))
+      (set-buffer (get-file-buffer tags-file))
+      (dolist (file (tags-table-files))
+        (let ((mtime (gethash file all-mtimes)))
+          (cond
+           ((null mtime)
+            (push file removed-files))
+           ((time-less-p tags-mtime mtime)
+            (push file changed-files)
+            (remhash file all-mtimes))
+           (t
+            (remhash file all-mtimes)))))
+      (maphash
+       (lambda (key _value)
+         (push key added-files))
+       all-mtimes)
+      (if (> (+ (length added-files)
+                (length changed-files)
+                (length removed-files))
+             100)
+          (progn
+            (message "etags-regen: Too many changes, falling back to full rescan")
+            (etags-regen--tags-cleanup))
+        (dolist (file (nconc removed-files changed-files))
+          (etags-regen--remove-tag file))
+        (when (or changed-files added-files)
+          (apply #'etags-regen--append-tags
+                 (nconc changed-files added-files)))
+        (when (or changed-files added-files removed-files)
+          (let ((save-silently t)
+                (message-log-max nil))
+            (save-buffer 0)))))))
+
+(defun etags-regen--maybe-generate ()
+  (let ((proj))
+    (when (and etags-regen--tags-root
+               (not (file-in-directory-p default-directory
+                                         etags-regen--tags-root)))
+      (etags-regen--tags-cleanup))
+    (when (and (not etags-regen--tags-root)
+               ;; If existing table is visited that's not generated by
+               ;; this mode, skip all functionality.
+               (not (or tags-file-name
+                        tags-table-list))
+               (file-exists-p (etags-regen--choose-tags-file
+                               (setq proj (project-current)))))
+      (message "Found existing tags table, refreshing...")
+      (etags-regen--refresh proj))
+    (when (and (not (or tags-file-name
+                        tags-table-list))
+               (setq proj (or proj (project-current))))
+      (message "Generating new tags table...")
+      (let ((start (time-to-seconds)))
+        (etags-regen--tags-generate proj)
+        (message "...done (%.2f s)" (- (time-to-seconds) start))))))
+
+(defun etags-regen--all-files (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         ;; TODO: Make the scanning more efficient, e.g. move the
+         ;; filtering by glob to project (project-files-filtered...).
+         (files (project-files proj))
+         (match-re (concat
+                    "\\."
+                    (regexp-opt etags-regen-file-extensions)
+                    "\\'")))
+    (cl-delete-if
+     (lambda (f) (or (not (string-match-p match-re f))
+                ;; FIXME: Handle etags-regen-ignores properly.
+                (string-match-p "/\\.#" f)
+                (cl-some (lambda (ignore) (string-search ignore f))
+                         etags-regen-ignores)))
+     files)))
+
+(defun etags-regen--tags-generate (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         (files (etags-regen--all-files proj))
+         (tags-file (etags-regen--choose-tags-file proj))
+         (ctags-p (etags-regen--ctags-p))
+         (command (format "%s %s %s - -o %s"
+                          etags-regen-program
+                          (mapconcat #'identity
+                                     (etags-regen--build-program-options ctags-p)
+                                     " ")
+                          ;; ctags's etags requires '-L' for stdin input.
+                          (if ctags-p "-L" "")
+                          tags-file)))
+    (with-temp-buffer
+      (mapc (lambda (f)
+              (insert f "\n"))
+            files)
+      (shell-command-on-region (point-min) (point-max) command
+                               nil nil etags-regen--errors-buffer-name t))
+    (etags-regen--visit-table tags-file root)))
+
+(defun etags-regen--visit-table (tags-file root)
+  ;; Invalidate the scanned tags after any change is written to disk.
+  (add-hook 'after-save-hook #'etags-regen--update-file)
+  (add-hook 'before-save-hook #'etags-regen--mark-as-new)
+  (setq etags-regen--tags-file tags-file
+        etags-regen--tags-root root)
+  (visit-tags-table etags-regen--tags-file))
+
+(defun etags-regen--ctags-p ()
+  (string-search "Ctags"
+                 (shell-command-to-string
+                  (format "%s --version" etags-regen-program))))
+
+(defun etags-regen--build-program-options (ctags-p)
+  (when (and etags-regen-regexp-alist ctags-p)
+    (user-error "etags-regen-regexp-alist is not supported with Ctags"))
+  (nconc
+   (mapcan
+    (lambda (group)
+      (mapcan
+       (lambda (lang)
+         (mapcar (lambda (regexp)
+                   (concat "--regex="
+                           (shell-quote-argument
+                            (format "{%s}%s" lang regexp))))
+                 (cdr group)))
+       (car group)))
+    etags-regen-regexp-alist)
+   etags-regen-program-options))
+
+(defun etags-regen--update-file ()
+  ;; TODO: Maybe only do this when Emacs is idle for a bit.  Or defer
+  ;; the updates and do them later in bursts when the table is used.
+  (let ((file-name buffer-file-name)
+        (tags-file-buf (and etags-regen--tags-root
+                            (get-file-buffer etags-regen--tags-file)))
+        pr should-scan)
+    (save-excursion
+      (when tags-file-buf
+        (cond
+         ((and etags-regen--new-file
+               (kill-local-variable 'etags-regen--new-file)
+               (setq pr (project-current))
+               (equal (project-root pr) etags-regen--tags-root)
+               (member file-name (project-files pr)))
+          (set-buffer tags-file-buf)
+          (setq should-scan t))
+         ((progn (set-buffer tags-file-buf)
+                 (etags-regen--remove-tag file-name))
+          (setq should-scan t))))
+      (when should-scan
+        (etags-regen--append-tags file-name)
+        (let ((save-silently t)
+              (message-log-max nil))
+          (save-buffer 0))))))
+
+(defun etags-regen--remove-tag (file-name)
+  (goto-char (point-min))
+  (when (search-forward (format "\f\n%s," file-name) nil t)
+    (let ((start (match-beginning 0)))
+      (search-forward "\f\n" nil 'move)
+      (let ((inhibit-read-only t))
+        (delete-region start
+                       (if (eobp)
+                           (point)
+                         (- (point) 2)))))
+    t))
+
+(defun etags-regen--append-tags (&rest file-names)
+  (goto-char (point-max))
+  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
+        (inhibit-read-only t))
+    ;; XXX: call-process is significantly faster, though.
+    ;; Like 10ms vs 20ms here.
+    (shell-command
+     (format "%s %s %s -o -"
+             etags-regen-program (mapconcat #'identity options " ")
+             (mapconcat #'identity file-names " "))
+     t etags-regen--errors-buffer-name))
+  ;; FIXME: Is there a better way to do this?
+  ;; Completion table is the only remaining place where the
+  ;; update is not incremental.
+  (setq-default tags-completion-table nil))
+
+(defun etags-regen--mark-as-new ()
+  (when (and etags-regen--tags-root
+             (not buffer-file-number))
+    (setq-local etags-regen--new-file t)))
+
+(defun etags-regen--tags-cleanup ()
+  (when etags-regen--tags-file
+    (let ((buffer (get-file-buffer etags-regen--tags-file)))
+      (and buffer
+           (kill-buffer buffer)))
+    (setq tags-file-name nil
+          tags-table-list nil
+          etags-regen--tags-file nil
+          etags-regen--tags-root nil))
+  (remove-hook 'after-save-hook #'etags-regen--update-file)
+  (remove-hook 'before-save-hook #'etags-regen--mark-as-new))
+
+(defvar etags-regen-mode-map (make-sparse-keymap))
+
+;;;###autoload
+(define-minor-mode etags-regen-mode
+  "Generate and update the tags automatically.
+
+This minor mode generates the tags table automatically based on
+the current project configuration, and later updates it as you
+edit the files and save the changes."
+  :global t
+  (if etags-regen-mode
+      (progn
+        (advice-add 'etags--xref-backend :before
+                    #'etags-regen--maybe-generate)
+        (advice-add 'tags-completion-at-point-function :before
+                    #'etags-regen--maybe-generate))
+    (advice-remove 'etags--xref-backend #'etags-regen--maybe-generate)
+    (advice-remove 'tags-completion-at-point-function #'etags-regen--maybe-generate)
+    (etags-regen--tags-cleanup)))
+
+(provide 'etags-regen)
+
+;;; etags-regen.el ends here

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

On 22/12/2023 01:37, Dmitry Gutov wrote:
> On 21/12/2023 18:46, Dmitry Gutov wrote:
>> See instead the patch attached to this bug report.
> 
> Here's an update, incorporating the feedback from here and there.

Fixed a typo in dir-locals and implemented better support for 
etags-regen-ignores (though with one omission).

To me it looks good to check in now.

[-- Attachment #2: etags-regen-v3.diff --]
[-- Type: text/x-patch, Size: 17646 bytes --]

diff --git a/.dir-locals.el b/.dir-locals.el
index e087aa89cd1..ce7febca851 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -8,6 +8,12 @@
          (vc-git-annotate-switches . "-w")
          (bug-reference-url-format . "https://debbugs.gnu.org/%s")
 	 (diff-add-log-use-relative-names . t)
+         (etags-regen-regexp-alist
+          .
+          ((("c" "objc") .
+            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
+             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
+         (etags-regen-ignores . ("test/manual/etags/"))
          (vc-prepare-patches-separately . nil)))
  (c-mode . ((c-file-style . "GNU")
             (c-noise-macro-names . ("INLINE" "NO_INLINE" "ATTRIBUTE_NO_SANITIZE_UNDEFINED"
diff --git a/etc/NEWS b/etc/NEWS
index 6df17aa3f0a..2ff3356d3c9 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1238,6 +1238,11 @@ the needs of users with red-green or blue-yellow color deficiency.
 The Info manual "(modus-themes) Top" describes the details and
 showcases all their customization options.
 
+** New global minor mode 'etags-regen-mode'.
+This minor mode generates the tags table automatically based on the
+current project configuration, and later updates it as you edit the
+files and save the changes.
+
 \f
 * Incompatible Lisp Changes in Emacs 30.1
 
diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
new file mode 100644
index 00000000000..88b730195c3
--- /dev/null
+++ b/lisp/progmodes/etags-regen.el
@@ -0,0 +1,411 @@
+;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2021, 2023 Free Software Foundation, Inc.
+
+;; Author: Dmitry Gutov <dmitry@gutov.dev>
+;; Keywords: tools
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Simple automatic tags generation with updates on save.
+;;
+;; The goal of this mode is to provide a feature that should be
+;; familiar to the users of certain lightweight programmer's editors,
+;; such as Sublime Text.  Which is "go to definition" with automatic
+;; indexing, added in ST3 (released in 2017).
+;;
+;; At the moment reindexing works off before/after-save-hook, but to
+;; handle more complex changes (e.g. the user switching to another
+;; branch from the terminal) we can look into plugging into something
+;; like `filenotify'.
+;;
+;; Note that this feature disables itself if the user has some tags
+;; table already visited (with `M-x visit-tags-table', or through an
+;; explicit prompt triggered by some feature that requires tags).
+
+;;; Code:
+
+(require 'cl-lib)
+
+(defgroup etags-regen nil
+  "Auto-(re)generating tags."
+  :group 'tools)
+
+(defvar etags-regen--tags-file nil)
+(defvar etags-regen--tags-root nil)
+(defvar etags-regen--new-file nil)
+
+(declare-function project-root "project")
+(declare-function project-files "project")
+(declare-function dired-glob-regexp "dired")
+
+(defcustom etags-regen-program (executable-find "etags")
+  "Name of the etags program.
+
+If you only have `ctags' installed, you can also set this to
+\"ctags -e\".  Some features might not be supported this way."
+  ;; Always having our 'etags' here would be easier, but we can't
+  ;; always rely on it being installed.  So it might be ctags's etags.
+  :type 'file)
+
+(defcustom etags-regen-tags-file "TAGS"
+  "Name of the tags file to create inside the project.
+
+This value should either be a simple file name (no directory
+specified), or a function that accepts a project root directory
+and returns a distinct file name for the tags file for it.  The
+latter option is most useful when you prefer to store the tag
+files somewhere outside -- e.g. in `temporary-file-directory'."
+  :type '(choice (string :tag "File name")
+                 (function :tag "Function that returns file name")))
+
+(defcustom etags-regen-program-options nil
+  "List of additional options to pass to the etags program."
+  :type '(repeat string))
+
+(defcustom etags-regen-regexp-alist nil
+  "Mapping of languages to additional regexps for tags.
+
+The value must be a list of conses (LANGUAGES . TAG-REGEXPS)
+where both car and cdr are lists of strings.
+
+Each language should be one of the recognized by etags, see
+`etags --help'.  Each tag regexp should be a string in the format
+as documented for the `--regex' arguments.
+
+We currently support only Emacs's etags program with this option."
+  :type '(repeat
+          (cons
+           :tag "Languages group"
+           (repeat (string :tag "Language name"))
+           (repeat (string :tag "Tag Regexp")))))
+
+;;;###autoload
+(put 'etags-regen-regexp-alist 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p
+             (lambda (group)
+               (and (consp group)
+                    (listp (car group))
+                    (listp (cdr group))
+                    (seq-every-p #'stringp (car group))
+                    (seq-every-p #'stringp (cdr group))))
+             value))))
+
+;; We have to list all extensions: etags falls back to Fortran
+;; when it cannot determine the type of the file.
+;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
+(defcustom etags-regen-file-extensions
+  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
+    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
+    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
+  "Code file extensions.
+
+File extensions to generate the tags for."
+  :type '(repeat (string :tag "File extension")))
+
+;;;###autoload
+(put 'etags-regen-file-extensions 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+;; FIXME: We don't support root anchoring yet.
+(defcustom etags-regen-ignores nil
+  "Additional ignore rules, in the format of `project-ignores'."
+  :type '(repeat
+          (string :tag "Glob to ignore")))
+
+;;;###autoload
+(put 'etags-regen-ignores 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
+
+(defun etags-regen--all-mtimes (proj)
+  (let ((files (etags-regen--all-files proj))
+        (mtimes (make-hash-table :test 'equal))
+        file-name-handler-alist)
+    (dolist (f files)
+      (condition-case nil
+          (puthash f
+                   (file-attribute-modification-time
+                    (file-attributes f))
+                   mtimes)
+        (file-missing nil)))
+    mtimes))
+
+(defun etags-regen--choose-tags-file (proj)
+  (if (functionp etags-regen-tags-file)
+      (funcall etags-regen-tags-file (project-root proj))
+    (expand-file-name etags-regen-tags-file (project-root proj))))
+
+(defun etags-regen--refresh (proj)
+  (save-excursion
+    (let* ((tags-file (etags-regen--choose-tags-file proj))
+           (tags-mtime (file-attribute-modification-time
+                        (file-attributes tags-file)))
+           (all-mtimes (etags-regen--all-mtimes proj))
+           added-files
+           changed-files
+           removed-files)
+      (etags-regen--visit-table tags-file (project-root proj))
+      (set-buffer (get-file-buffer tags-file))
+      (dolist (file (tags-table-files))
+        (let ((mtime (gethash file all-mtimes)))
+          (cond
+           ((null mtime)
+            (push file removed-files))
+           ((time-less-p tags-mtime mtime)
+            (push file changed-files)
+            (remhash file all-mtimes))
+           (t
+            (remhash file all-mtimes)))))
+      (maphash
+       (lambda (key _value)
+         (push key added-files))
+       all-mtimes)
+      (if (> (+ (length added-files)
+                (length changed-files)
+                (length removed-files))
+             100)
+          (progn
+            (message "etags-regen: Too many changes, falling back to full rescan")
+            (etags-regen--tags-cleanup))
+        (dolist (file (nconc removed-files changed-files))
+          (etags-regen--remove-tag file))
+        (when (or changed-files added-files)
+          (apply #'etags-regen--append-tags
+                 (nconc changed-files added-files)))
+        (when (or changed-files added-files removed-files)
+          (let ((save-silently t)
+                (message-log-max nil))
+            (save-buffer 0)))))))
+
+(defun etags-regen--maybe-generate ()
+  (let ((proj))
+    (when (and etags-regen--tags-root
+               (not (file-in-directory-p default-directory
+                                         etags-regen--tags-root)))
+      (etags-regen--tags-cleanup))
+    (when (and (not etags-regen--tags-root)
+               ;; If existing table is visited that's not generated by
+               ;; this mode, skip all functionality.
+               (not (or tags-file-name
+                        tags-table-list))
+               (file-exists-p (etags-regen--choose-tags-file
+                               (setq proj (project-current)))))
+      (message "Found existing tags table, refreshing...")
+      (etags-regen--refresh proj))
+    (when (and (not (or tags-file-name
+                        tags-table-list))
+               (setq proj (or proj (project-current))))
+      (message "Generating new tags table...")
+      (let ((start (time-to-seconds)))
+        (etags-regen--tags-generate proj)
+        (message "...done (%.2f s)" (- (time-to-seconds) start))))))
+
+(defun etags-regen--all-files (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         ;; TODO: Make the scanning more efficient, e.g. move the
+         ;; filtering by glob to project (project-files-filtered...).
+         (files (project-files proj))
+         (match-re (concat
+                    "\\."
+                    (regexp-opt etags-regen-file-extensions)
+                    "\\'"))
+         (ir-start (1- (length root)))
+         (ignores-regexps
+          (mapcar #'etags-regen--ignore-regexp
+                  etags-regen-ignores)))
+    (cl-delete-if
+     (lambda (f) (or (not (string-match-p match-re f))
+                (string-match-p "/\\.#" f)
+                (cl-some (lambda (ignore) (string-match ignore f ir-start))
+                         ignores-regexps)))
+     files)))
+
+(defun etags-regen--ignore-regexp (ignore)
+  (require 'dired)
+  ;; It's somewhat brittle to rely on Dired.
+  (let ((re (dired-glob-regexp ignore)))
+    ;; We could implement root anchoring here, but \\= doesn't work in
+    ;; string-match :-(.
+    (concat (unless (eq ?/ (aref re 3)) "/")
+            ;; Cutting off the anchors.
+            (substring re 2 (- (length re) 2))
+            (unless (eq ?/ (aref re (- (length re) 3)))
+              ;; Either match a full name segment, or eos.
+              "\\(?:/\\|\\'\\)"))))
+
+(defun etags-regen--tags-generate (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         (files (etags-regen--all-files proj))
+         (tags-file (etags-regen--choose-tags-file proj))
+         (ctags-p (etags-regen--ctags-p))
+         (command (format "%s %s %s - -o %s"
+                          etags-regen-program
+                          (mapconcat #'identity
+                                     (etags-regen--build-program-options ctags-p)
+                                     " ")
+                          ;; ctags's etags requires '-L' for stdin input.
+                          (if ctags-p "-L" "")
+                          tags-file)))
+    (with-temp-buffer
+      (mapc (lambda (f)
+              (insert f "\n"))
+            files)
+      (shell-command-on-region (point-min) (point-max) command
+                               nil nil etags-regen--errors-buffer-name t))
+    (etags-regen--visit-table tags-file root)))
+
+(defun etags-regen--visit-table (tags-file root)
+  ;; Invalidate the scanned tags after any change is written to disk.
+  (add-hook 'after-save-hook #'etags-regen--update-file)
+  (add-hook 'before-save-hook #'etags-regen--mark-as-new)
+  (setq etags-regen--tags-file tags-file
+        etags-regen--tags-root root)
+  (visit-tags-table etags-regen--tags-file))
+
+(defun etags-regen--ctags-p ()
+  (string-search "Ctags"
+                 (shell-command-to-string
+                  (format "%s --version" etags-regen-program))))
+
+(defun etags-regen--build-program-options (ctags-p)
+  (when (and etags-regen-regexp-alist ctags-p)
+    (user-error "etags-regen-regexp-alist is not supported with Ctags"))
+  (nconc
+   (mapcan
+    (lambda (group)
+      (mapcan
+       (lambda (lang)
+         (mapcar (lambda (regexp)
+                   (concat "--regex="
+                           (shell-quote-argument
+                            (format "{%s}%s" lang regexp))))
+                 (cdr group)))
+       (car group)))
+    etags-regen-regexp-alist)
+   etags-regen-program-options))
+
+(defun etags-regen--update-file ()
+  ;; TODO: Maybe only do this when Emacs is idle for a bit.  Or defer
+  ;; the updates and do them later in bursts when the table is used.
+  (let* ((file-name buffer-file-name)
+         (tags-file-buf (and etags-regen--tags-root
+                             (get-file-buffer etags-regen--tags-file)))
+         (relname (concat "/" (file-relative-name file-name
+                                                  etags-regen--tags-root)))
+         (ignores etags-regen-ignores)
+         pr should-scan)
+    (save-excursion
+      (when tags-file-buf
+        (cond
+         ((and etags-regen--new-file
+               (kill-local-variable 'etags-regen--new-file)
+               (setq pr (project-current))
+               (equal (project-root pr) etags-regen--tags-root)
+               (member file-name (project-files pr)))
+          (set-buffer tags-file-buf)
+          (setq should-scan t))
+         ((progn (set-buffer tags-file-buf)
+                 (etags-regen--remove-tag file-name))
+          (setq should-scan t))))
+      (when (and should-scan
+                 (not (cl-some
+                       (lambda (ignore)
+                         (string-match-p
+                          (etags-regen--ignore-regexp ignore)
+                          relname))
+                       ignores)))
+        (etags-regen--append-tags file-name)
+        (let ((save-silently t)
+              (message-log-max nil))
+          (save-buffer 0))))))
+
+(defun etags-regen--remove-tag (file-name)
+  (goto-char (point-min))
+  (when (search-forward (format "\f\n%s," file-name) nil t)
+    (let ((start (match-beginning 0)))
+      (search-forward "\f\n" nil 'move)
+      (let ((inhibit-read-only t))
+        (delete-region start
+                       (if (eobp)
+                           (point)
+                         (- (point) 2)))))
+    t))
+
+(defun etags-regen--append-tags (&rest file-names)
+  (goto-char (point-max))
+  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
+        (inhibit-read-only t))
+    ;; XXX: call-process is significantly faster, though.
+    ;; Like 10ms vs 20ms here.
+    (shell-command
+     (format "%s %s %s -o -"
+             etags-regen-program (mapconcat #'identity options " ")
+             (mapconcat #'identity file-names " "))
+     t etags-regen--errors-buffer-name))
+  ;; FIXME: Is there a better way to do this?
+  ;; Completion table is the only remaining place where the
+  ;; update is not incremental.
+  (setq-default tags-completion-table nil))
+
+(defun etags-regen--mark-as-new ()
+  (when (and etags-regen--tags-root
+             (not buffer-file-number))
+    (setq-local etags-regen--new-file t)))
+
+(defun etags-regen--tags-cleanup ()
+  (when etags-regen--tags-file
+    (let ((buffer (get-file-buffer etags-regen--tags-file)))
+      (and buffer
+           (kill-buffer buffer)))
+    (tags-reset-tags-tables)
+    (setq tags-file-name nil
+          tags-table-list nil
+          etags-regen--tags-file nil
+          etags-regen--tags-root nil))
+  (remove-hook 'after-save-hook #'etags-regen--update-file)
+  (remove-hook 'before-save-hook #'etags-regen--mark-as-new))
+
+(defvar etags-regen-mode-map (make-sparse-keymap))
+
+;;;###autoload
+(define-minor-mode etags-regen-mode
+  "Generate and update the tags automatically.
+
+This minor mode generates the tags table automatically based on
+the current project configuration, and later updates it as you
+edit the files and save the changes."
+  :global t
+  (if etags-regen-mode
+      (progn
+        (advice-add 'etags--xref-backend :before
+                    #'etags-regen--maybe-generate)
+        (advice-add 'tags-completion-at-point-function :before
+                    #'etags-regen--maybe-generate))
+    (advice-remove 'etags--xref-backend #'etags-regen--maybe-generate)
+    (advice-remove 'tags-completion-at-point-function #'etags-regen--maybe-generate)
+    (etags-regen--tags-cleanup)))
+
+(provide 'etags-regen)
+
+;;; etags-regen.el ends here

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Sun, 24 Dec 2023 03:43:25 +0200
> From: Dmitry Gutov <dmitry@gutov.dev>
> Cc: 67687@debbugs.gnu.org, eskinjp@gmail.com, stefankangas@gmail.com
> 
> On 22/12/2023 01:37, Dmitry Gutov wrote:
> > On 21/12/2023 18:46, Dmitry Gutov wrote:
> >> See instead the patch attached to this bug report.
> > 
> > Here's an update, incorporating the feedback from here and there.
> 
> Fixed a typo in dir-locals and implemented better support for 
> etags-regen-ignores (though with one omission).
> 
> To me it looks good to check in now.

Thanks, I have a few minor comments.

> +(defcustom etags-regen-program (executable-find "etags")
> +  "Name of the etags program.
> +
> +If you only have `ctags' installed, you can also set this to
> +\"ctags -e\".  Some features might not be supported this way."
> +  ;; Always having our 'etags' here would be easier, but we can't
> +  ;; always rely on it being installed.  So it might be ctags's etags.
> +  :type 'file)

Please add :version tags to all the defcustoms you add.

> +(defcustom etags-regen-tags-file "TAGS"
> +  "Name of the tags file to create inside the project.

This and the other defcustom's here should say in the first line of
the doc string that they are for etags-regen-mode.  This will help
discoverability and also produce a more helpful display with the
various apropos commands.

> +This value should either be a simple file name (no directory
   ^^^^^^^^^^
"The value" or just "Value".

> +specified), or a function that accepts a project root directory
> +and returns a distinct file name for the tags file for it.

That function should also return only a file name without leading
directories, right?  If so, the text should be more explicit about
that.  For example:

  Value should be either a string specifying a file name without
  leading directories, or or a function that accepts a project's root
  directory and returns such a file name, to be used as the tags file
  for that project.

>                                                              The
> +latter option is most useful when you prefer to store the tag
          ^^^^^^
Using "option" in a doc string of a user option might be ambiguous.  I
suggest to use "alternative" or "possibility" instead.

> +files somewhere outside -- e.g. in `temporary-file-directory'."

So the function could return a file name _with_ leading directories?

> +;;;###autoload
> +(put 'etags-regen-file-extensions 'safe-local-variable
> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))

Why not use list-of-strings-p here?

> +;;;###autoload
> +(put 'etags-regen-ignores 'safe-local-variable
> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))

And here.

> +      (if (> (+ (length added-files)
> +                (length changed-files)
> +                (length removed-files))
> +             100)
> +          (progn
> +            (message "etags-regen: Too many changes, falling back to full rescan")

Should the magic 100 value be a defvar, not a hard-coded constant?

> +(defun etags-regen--maybe-generate ()
> +  (let ((proj))

Would

     (let (proj)

do here?  IOW, why the extra pair of parens?

> +     (lambda (f) (or (not (string-match-p match-re f))
> +                (string-match-p "/\\.#" f)

Is that '/' there to detect regexps for absolute file names?  If so,
that won't work for Windows.

> +(defun etags-regen--ignore-regexp (ignore)
> +  (require 'dired)
> +  ;; It's somewhat brittle to rely on Dired.
> +  (let ((re (dired-glob-regexp ignore)))
> +    ;; We could implement root anchoring here, but \\= doesn't work in
> +    ;; string-match :-(.
> +    (concat (unless (eq ?/ (aref re 3)) "/")
> +            ;; Cutting off the anchors.
> +            (substring re 2 (- (length re) 2))
> +            (unless (eq ?/ (aref re (- (length re) 3)))
> +              ;; Either match a full name segment, or eos.
> +              "\\(?:/\\|\\'\\)"))))

Same here: what is the purpose of comparisons with a slash?  I think
we need some more comments there explaining the logic of the code.

> +(defun etags-regen--append-tags (&rest file-names)
> +  (goto-char (point-max))
> +  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
> +        (inhibit-read-only t))
> +    ;; XXX: call-process is significantly faster, though.
> +    ;; Like 10ms vs 20ms here.
> +    (shell-command
> +     (format "%s %s %s -o -"
> +             etags-regen-program (mapconcat #'identity options " ")
> +             (mapconcat #'identity file-names " "))
> +     t etags-regen--errors-buffer-name))

Should we indeed use call-process?

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 6236 bytes --]

On 28/12/2023 11:30, Eli Zaretskii wrote:
>> Date: Sun, 24 Dec 2023 03:43:25 +0200
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> Cc: 67687@debbugs.gnu.org, eskinjp@gmail.com, stefankangas@gmail.com
>>
>> On 22/12/2023 01:37, Dmitry Gutov wrote:
>>> On 21/12/2023 18:46, Dmitry Gutov wrote:
>>>> See instead the patch attached to this bug report.
>>>
>>> Here's an update, incorporating the feedback from here and there.
>>
>> Fixed a typo in dir-locals and implemented better support for
>> etags-regen-ignores (though with one omission).
>>
>> To me it looks good to check in now.
> 
> Thanks, I have a few minor comments.

Attaching the next version, which addresses most but not all comments of 
yours and Stefan's.

>> +(defcustom etags-regen-program (executable-find "etags")
>> +  "Name of the etags program.
>> +
>> +If you only have `ctags' installed, you can also set this to
>> +\"ctags -e\".  Some features might not be supported this way."
>> +  ;; Always having our 'etags' here would be easier, but we can't
>> +  ;; always rely on it being installed.  So it might be ctags's etags.
>> +  :type 'file)
> 
> Please add :version tags to all the defcustoms you add.

Done.

>> +(defcustom etags-regen-tags-file "TAGS"
>> +  "Name of the tags file to create inside the project.
> 
> This and the other defcustom's here should say in the first line of
> the doc string that they are for etags-regen-mode.  This will help
> discoverability and also produce a more helpful display with the
> various apropos commands.

I've tried, but it seems hard to fit into most of them while keeping to 
the requisite max number of columns. Only managed to fit that into 
etags-regen-program and etags-regen-file-extensions.

TBH, most of the time it would seem superfluous, given the namespaced 
names. But it's probably good to mention in 'etags-regen-program', on 
balance.

>> +This value should either be a simple file name (no directory
>     ^^^^^^^^^^
> "The value" or just "Value".

Ok.

>> +specified), or a function that accepts a project root directory
>> +and returns a distinct file name for the tags file for it.
> 
> That function should also return only a file name without leading
> directories, right?  If so, the text should be more explicit about
> that.  For example:
> 
>    Value should be either a string specifying a file name without
>    leading directories, or or a function that accepts a project's root
>    directory and returns such a file name, to be used as the tags file
>    for that project.
> 
>>                                                               The
>> +latter option is most useful when you prefer to store the tag
>            ^^^^^^
> Using "option" in a doc string of a user option might be ambiguous.  I
> suggest to use "alternative" or "possibility" instead.

Done.

>> +files somewhere outside -- e.g. in `temporary-file-directory'."
> 
> So the function could return a file name _with_ leading directories?

Indeed. So I added "absolute" to the docstring.

We could allow it to return a nondirectory name too, if somebody finds a 
use case.

>> +;;;###autoload
>> +(put 'etags-regen-file-extensions 'safe-local-variable
>> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
> 
> Why not use list-of-strings-p here?

Again, that "core ELPA" consideration. We could deploy this feature to a 
number of released Emacs versions, if we don't introduce such dependencies.

>> +;;;###autoload
>> +(put 'etags-regen-ignores 'safe-local-variable
>> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
> 
> And here.
> 
>> +      (if (> (+ (length added-files)
>> +                (length changed-files)
>> +                (length removed-files))
>> +             100)
>> +          (progn
>> +            (message "etags-regen: Too many changes, falling back to full rescan")
> 
> Should the magic 100 value be a defvar, not a hard-coded constant?

Moved it to a defvar.

>> +(defun etags-regen--maybe-generate ()
>> +  (let ((proj))
> 
> Would
> 
>       (let (proj)
> 
> do here?  IOW, why the extra pair of parens?

Altered.

>> +     (lambda (f) (or (not (string-match-p match-re f))
>> +                (string-match-p "/\\.#" f)
> 
> Is that '/' there to detect regexps for absolute file names?  If so,
> that won't work for Windows.

It's to detect backup files.

>> +(defun etags-regen--ignore-regexp (ignore)
>> +  (require 'dired)
>> +  ;; It's somewhat brittle to rely on Dired.
>> +  (let ((re (dired-glob-regexp ignore)))
>> +    ;; We could implement root anchoring here, but \\= doesn't work in
>> +    ;; string-match :-(.
>> +    (concat (unless (eq ?/ (aref re 3)) "/")
>> +            ;; Cutting off the anchors.
>> +            (substring re 2 (- (length re) 2))
>> +            (unless (eq ?/ (aref re (- (length re) 3)))
>> +              ;; Either match a full name segment, or eos.
>> +              "\\(?:/\\|\\'\\)"))))
> 
> Same here: what is the purpose of comparisons with a slash?  I think
> we need some more comments there explaining the logic of the code.

We compare with a slash to see whether the glob was matching against a 
directory (in which case it's already anchored to the name of a file 
name segment), otherwise we add such anchoring to either the end of a 
file name segment or eos (thus allowing a glob match both directory 
names and file names).

Added a shorter comment saying the same.

>> +(defun etags-regen--append-tags (&rest file-names)
>> +  (goto-char (point-max))
>> +  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
>> +        (inhibit-read-only t))
>> +    ;; XXX: call-process is significantly faster, though.
>> +    ;; Like 10ms vs 20ms here.
>> +    (shell-command
>> +     (format "%s %s %s -o -"
>> +             etags-regen-program (mapconcat #'identity options " ")
>> +             (mapconcat #'identity file-names " "))
>> +     t etags-regen--errors-buffer-name))
> 
> Should we indeed use call-process?

Something for later improvement.

Looking at the code, I believe I decided to use 'shell-command' for the 
first version because of how easy it makes to output stderr to a 
separate buffer. call-process only offers writing them to a file.

[-- Attachment #2: etags-regen-v4.diff --]
[-- Type: text/x-patch, Size: 17962 bytes --]

diff --git a/.dir-locals.el b/.dir-locals.el
index e087aa89cd1..ce7febca851 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -8,6 +8,12 @@
          (vc-git-annotate-switches . "-w")
          (bug-reference-url-format . "https://debbugs.gnu.org/%s")
 	 (diff-add-log-use-relative-names . t)
+         (etags-regen-regexp-alist
+          .
+          ((("c" "objc") .
+            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
+             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
+         (etags-regen-ignores . ("test/manual/etags/"))
          (vc-prepare-patches-separately . nil)))
  (c-mode . ((c-file-style . "GNU")
             (c-noise-macro-names . ("INLINE" "NO_INLINE" "ATTRIBUTE_NO_SANITIZE_UNDEFINED"
diff --git a/etc/NEWS b/etc/NEWS
index f82564946b7..6d6bca187de 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1243,6 +1243,11 @@ the needs of users with red-green or blue-yellow color deficiency.
 The Info manual "(modus-themes) Top" describes the details and
 showcases all their customization options.
 
+** New global minor mode 'etags-regen-mode'.
+This minor mode generates the tags table automatically based on the
+current project configuration, and later updates it as you edit the
+files and save the changes.
+
 \f
 * Incompatible Lisp Changes in Emacs 30.1
 
diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
new file mode 100644
index 00000000000..22ae9ce4f0a
--- /dev/null
+++ b/lisp/progmodes/etags-regen.el
@@ -0,0 +1,420 @@
+;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2021-2023 Free Software Foundation, Inc.
+
+;; Author: Dmitry Gutov <dmitry@gutov.dev>
+;; Keywords: tools
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Simple automatic tags generation with updates on save.
+;;
+;; This mode provides automatic indexing for Emacs "go to definition"
+;; feature, the `xref-go-forward' command (bound to `M-.' by default).
+;;
+;; At the moment reindexing works off before/after-save-hook, but to
+;; handle more complex changes (e.g. the user switching to another
+;; branch from the terminal) we can look into plugging into something
+;; like `filenotify'.
+;;
+;; Note that this feature disables itself if the user has some tags
+;; table already visited (with `M-x visit-tags-table', or through an
+;; explicit prompt triggered by some feature that requires tags).
+
+;;; Code:
+
+(require 'cl-lib)
+
+(defgroup etags-regen nil
+  "Auto-(re)generating tags."
+  :group 'tools)
+
+(defvar etags-regen--tags-file nil)
+(defvar etags-regen--tags-root nil)
+(defvar etags-regen--new-file nil)
+
+(declare-function project-root "project")
+(declare-function project-files "project")
+(declare-function dired-glob-regexp "dired")
+
+(defcustom etags-regen-program (executable-find "etags")
+  "Name of the etags program used by `etags-regen-mode'.
+
+If you only have `ctags' installed, you can also set this to
+\"ctags -e\".  Some features might not be supported this way."
+  ;; Always having our 'etags' here would be easier, but we can't
+  ;; always rely on it being installed.  So it might be ctags's etags.
+  :type 'file
+  :version "30.1")
+
+(defcustom etags-regen-tags-file "TAGS"
+  "Name of the tags file to create inside the project.
+
+The value should either be a simple file name (no directory
+specified), or a function that accepts a project root directory
+and returns a distinct absolute file name for its tags file.  The
+latter possibility is useful when you prefer to store the tag
+files somewhere outside -- e.g. in `temporary-file-directory'."
+  :type '(choice (string :tag "File name")
+                 (function :tag "Function that returns file name"))
+  :version "30.1")
+
+(defcustom etags-regen-program-options nil
+  "List of additional options to pass to the etags program."
+  :type '(repeat string)
+  :version "30.1")
+
+(defcustom etags-regen-regexp-alist nil
+  "Mapping of languages to additional regexps for tags.
+
+The value must be a list of conses (LANGUAGES . TAG-REGEXPS)
+where both car and cdr are lists of strings.
+
+Each language should be one of the recognized by etags, see
+`etags --help'.  Each tag regexp should be a string in the format
+as documented for the `--regex' arguments.
+
+We currently support only Emacs's etags program with this option."
+  :type '(repeat
+          (cons
+           :tag "Languages group"
+           (repeat (string :tag "Language name"))
+           (repeat (string :tag "Tag Regexp"))))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-regexp-alist 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p
+             (lambda (group)
+               (and (consp group)
+                    (listp (car group))
+                    (listp (cdr group))
+                    (seq-every-p #'stringp (car group))
+                    (seq-every-p #'stringp (cdr group))))
+             value))))
+
+;; We have to list all extensions: etags falls back to Fortran
+;; when it cannot determine the type of the file.
+;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
+(defcustom etags-regen-file-extensions
+  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
+    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
+    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
+  "Code file extensions for `etags-regen-mode'.
+
+File extensions to generate the tags for."
+  :type '(repeat (string :tag "File extension"))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-file-extensions 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+;; FIXME: We don't support root anchoring yet.
+(defcustom etags-regen-ignores nil
+  "Additional ignore rules, in the format of `project-ignores'."
+  :type '(repeat
+          (string :tag "Glob to ignore"))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-ignores 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
+
+(defvar etags-regen--rescan-files-limit 100)
+
+(defun etags-regen--all-mtimes (proj)
+  (let ((files (etags-regen--all-files proj))
+        (mtimes (make-hash-table :test 'equal))
+        file-name-handler-alist)
+    (dolist (f files)
+      (condition-case nil
+          (puthash f
+                   (file-attribute-modification-time
+                    (file-attributes f))
+                   mtimes)
+        (file-missing nil)))
+    mtimes))
+
+(defun etags-regen--choose-tags-file (proj)
+  (if (functionp etags-regen-tags-file)
+      (funcall etags-regen-tags-file (project-root proj))
+    (expand-file-name etags-regen-tags-file (project-root proj))))
+
+(defun etags-regen--refresh (proj)
+  (save-excursion
+    (let* ((tags-file (etags-regen--choose-tags-file proj))
+           (tags-mtime (file-attribute-modification-time
+                        (file-attributes tags-file)))
+           (all-mtimes (etags-regen--all-mtimes proj))
+           added-files
+           changed-files
+           removed-files)
+      (etags-regen--visit-table tags-file (project-root proj))
+      (set-buffer (get-file-buffer tags-file))
+      (dolist (file (tags-table-files))
+        (let ((mtime (gethash file all-mtimes)))
+          (cond
+           ((null mtime)
+            (push file removed-files))
+           ((time-less-p tags-mtime mtime)
+            (push file changed-files)
+            (remhash file all-mtimes))
+           (t
+            (remhash file all-mtimes)))))
+      (maphash
+       (lambda (key _value)
+         (push key added-files))
+       all-mtimes)
+      (if (> (+ (length added-files)
+                (length changed-files)
+                (length removed-files))
+             etags-regen--rescan-files-limit)
+          (progn
+            (message "etags-regen: Too many changes, falling back to full rescan")
+            (etags-regen--tags-cleanup))
+        (dolist (file (nconc removed-files changed-files))
+          (etags-regen--remove-tag file))
+        (when (or changed-files added-files)
+          (apply #'etags-regen--append-tags
+                 (nconc changed-files added-files)))
+        (when (or changed-files added-files removed-files)
+          (let ((save-silently t)
+                (message-log-max nil))
+            (save-buffer 0)))))))
+
+(defun etags-regen--maybe-generate ()
+  (let (proj)
+    (when (and etags-regen--tags-root
+               (not (file-in-directory-p default-directory
+                                         etags-regen--tags-root)))
+      (etags-regen--tags-cleanup))
+    (when (and (not etags-regen--tags-root)
+               ;; If existing table is visited that's not generated by
+               ;; this mode, skip all functionality.
+               (not (or tags-file-name
+                        tags-table-list))
+               (file-exists-p (etags-regen--choose-tags-file
+                               (setq proj (project-current)))))
+      (message "Found existing tags table, refreshing...")
+      (etags-regen--refresh proj))
+    (when (and (not (or tags-file-name
+                        tags-table-list))
+               (setq proj (or proj (project-current))))
+      (message "Generating new tags table...")
+      (let ((start (time-to-seconds)))
+        (etags-regen--tags-generate proj)
+        (message "...done (%.2f s)" (- (time-to-seconds) start))))))
+
+(defun etags-regen--all-files (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         ;; TODO: Make the scanning more efficient, e.g. move the
+         ;; filtering by glob to project (project-files-filtered...).
+         (files (project-files proj))
+         (match-re (concat
+                    "\\."
+                    (regexp-opt etags-regen-file-extensions)
+                    "\\'"))
+         (ir-start (1- (length root)))
+         (ignores-regexps
+          (mapcar #'etags-regen--ignore-regexp
+                  etags-regen-ignores)))
+    (cl-delete-if
+     (lambda (f) (or (not (string-match-p match-re f))
+                (string-match-p "/\\.#" f)
+                (cl-some (lambda (ignore) (string-match ignore f ir-start))
+                         ignores-regexps)))
+     files)))
+
+(defun etags-regen--ignore-regexp (ignore)
+  (require 'dired)
+  ;; It's somewhat brittle to rely on Dired.
+  (let ((re (dired-glob-regexp ignore)))
+    ;; We could implement root anchoring here, but \\= doesn't work in
+    ;; string-match :-(.
+    (concat (unless (eq ?/ (aref re 3)) "/")
+            ;; Cutting off the anchors.
+            (substring re 2 (- (length re) 2))
+            ;; This way we allow a glob to match against a directory
+            ;; name, or a file name.  And when it ends with / already,
+            ;; no need to add the anchoring.
+            (unless (eq ?/ (aref re (- (length re) 3)))
+              ;; Either match a full name segment, or eos.
+              "\\(?:/\\|\\'\\)"))))
+
+(defun etags-regen--tags-generate (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         (files (etags-regen--all-files proj))
+         (tags-file (etags-regen--choose-tags-file proj))
+         (ctags-p (etags-regen--ctags-p))
+         (command (format "%s %s %s - -o %s"
+                          etags-regen-program
+                          (mapconcat #'identity
+                                     (etags-regen--build-program-options ctags-p)
+                                     " ")
+                          ;; ctags's etags requires '-L' for stdin input.
+                          (if ctags-p "-L" "")
+                          tags-file)))
+    (with-temp-buffer
+      (mapc (lambda (f)
+              (insert f "\n"))
+            files)
+      (shell-command-on-region (point-min) (point-max) command
+                               nil nil etags-regen--errors-buffer-name t))
+    (etags-regen--visit-table tags-file root)))
+
+(defun etags-regen--visit-table (tags-file root)
+  ;; Invalidate the scanned tags after any change is written to disk.
+  (add-hook 'after-save-hook #'etags-regen--update-file)
+  (add-hook 'before-save-hook #'etags-regen--mark-as-new)
+  (setq etags-regen--tags-file tags-file
+        etags-regen--tags-root root)
+  (visit-tags-table etags-regen--tags-file))
+
+(defun etags-regen--ctags-p ()
+  (string-search "Ctags"
+                 (shell-command-to-string
+                  (format "%s --version" etags-regen-program))))
+
+(defun etags-regen--build-program-options (ctags-p)
+  (when (and etags-regen-regexp-alist ctags-p)
+    (user-error "etags-regen-regexp-alist is not supported with Ctags"))
+  (nconc
+   (mapcan
+    (lambda (group)
+      (mapcan
+       (lambda (lang)
+         (mapcar (lambda (regexp)
+                   (concat "--regex="
+                           (shell-quote-argument
+                            (format "{%s}%s" lang regexp))))
+                 (cdr group)))
+       (car group)))
+    etags-regen-regexp-alist)
+   etags-regen-program-options))
+
+(defun etags-regen--update-file ()
+  ;; TODO: Maybe only do this when Emacs is idle for a bit.  Or defer
+  ;; the updates and do them later in bursts when the table is used.
+  (let* ((file-name buffer-file-name)
+         (tags-file-buf (and etags-regen--tags-root
+                             (get-file-buffer etags-regen--tags-file)))
+         (relname (concat "/" (file-relative-name file-name
+                                                  etags-regen--tags-root)))
+         (ignores etags-regen-ignores)
+         pr should-scan)
+    (save-excursion
+      (when tags-file-buf
+        (cond
+         ((and etags-regen--new-file
+               (kill-local-variable 'etags-regen--new-file)
+               (setq pr (project-current))
+               (equal (project-root pr) etags-regen--tags-root)
+               (member file-name (project-files pr)))
+          (set-buffer tags-file-buf)
+          (setq should-scan t))
+         ((progn (set-buffer tags-file-buf)
+                 (etags-regen--remove-tag file-name))
+          (setq should-scan t))))
+      (when (and should-scan
+                 (not (cl-some
+                       (lambda (ignore)
+                         (string-match-p
+                          (etags-regen--ignore-regexp ignore)
+                          relname))
+                       ignores)))
+        (etags-regen--append-tags file-name)
+        (let ((save-silently t)
+              (message-log-max nil))
+          (save-buffer 0))))))
+
+(defun etags-regen--remove-tag (file-name)
+  (goto-char (point-min))
+  (when (search-forward (format "\f\n%s," file-name) nil t)
+    (let ((start (match-beginning 0)))
+      (search-forward "\f\n" nil 'move)
+      (let ((inhibit-read-only t))
+        (delete-region start
+                       (if (eobp)
+                           (point)
+                         (- (point) 2)))))
+    t))
+
+(defun etags-regen--append-tags (&rest file-names)
+  (goto-char (point-max))
+  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
+        (inhibit-read-only t))
+    ;; XXX: call-process is significantly faster, though.
+    ;; Like 10ms vs 20ms here.
+    (shell-command
+     (format "%s %s %s -o -"
+             etags-regen-program (mapconcat #'identity options " ")
+             (mapconcat #'identity file-names " "))
+     t etags-regen--errors-buffer-name))
+  ;; FIXME: Is there a better way to do this?
+  ;; Completion table is the only remaining place where the
+  ;; update is not incremental.
+  (setq-default tags-completion-table nil))
+
+(defun etags-regen--mark-as-new ()
+  (when (and etags-regen--tags-root
+             (not buffer-file-number))
+    (setq-local etags-regen--new-file t)))
+
+(defun etags-regen--tags-cleanup ()
+  (when etags-regen--tags-file
+    (let ((buffer (get-file-buffer etags-regen--tags-file)))
+      (and buffer
+           (kill-buffer buffer)))
+    (tags-reset-tags-tables)
+    (setq tags-file-name nil
+          tags-table-list nil
+          etags-regen--tags-file nil
+          etags-regen--tags-root nil))
+  (remove-hook 'after-save-hook #'etags-regen--update-file)
+  (remove-hook 'before-save-hook #'etags-regen--mark-as-new))
+
+(defvar etags-regen-mode-map (make-sparse-keymap))
+
+;;;###autoload
+(define-minor-mode etags-regen-mode
+  "Generate and update the tags automatically.
+
+This minor mode generates the tags table automatically based on
+the current project configuration, and later updates it as you
+edit the files and save the changes."
+  :global t
+  (if etags-regen-mode
+      (progn
+        (advice-add 'etags--xref-backend :before
+                    #'etags-regen--maybe-generate)
+        (advice-add 'tags-completion-at-point-function :before
+                    #'etags-regen--maybe-generate))
+    (advice-remove 'etags--xref-backend #'etags-regen--maybe-generate)
+    (advice-remove 'tags-completion-at-point-function #'etags-regen--maybe-generate)
+    (etags-regen--tags-cleanup)))
+
+(provide 'etags-regen)
+
+;;; etags-regen.el ends here

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Sat, 30 Dec 2023 05:05:01 +0200
> Cc: 67687@debbugs.gnu.org, eskinjp@gmail.com, stefankangas@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> +(defcustom etags-regen-tags-file "TAGS"
> >> +  "Name of the tags file to create inside the project.
> > 
> > This and the other defcustom's here should say in the first line of
> > the doc string that they are for etags-regen-mode.  This will help
> > discoverability and also produce a more helpful display with the
> > various apropos commands.
> 
> I've tried, but it seems hard to fit into most of them while keeping to 
> the requisite max number of columns. Only managed to fit that into 
> etags-regen-program and etags-regen-file-extensions.
> 
> TBH, most of the time it would seem superfluous, given the namespaced 
> names. But it's probably good to mention in 'etags-regen-program', on 
> balance.

I suggest some minor improvements in this area below.

> >> +;;;###autoload
> >> +(put 'etags-regen-file-extensions 'safe-local-variable
> >> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
> > 
> > Why not use list-of-strings-p here?
> 
> Again, that "core ELPA" consideration. We could deploy this feature to a 
> number of released Emacs versions, if we don't introduce such dependencies.

Isn't this covered by the compat package on ELPA?  If not, I think it
should be.

> >> +     (lambda (f) (or (not (string-match-p match-re f))
> >> +                (string-match-p "/\\.#" f)
> > 
> > Is that '/' there to detect regexps for absolute file names?  If so,
> > that won't work for Windows.
> 
> It's to detect backup files.

Can you add a comment there to that effect?

> 
> >> +(defun etags-regen--ignore-regexp (ignore)
> >> +  (require 'dired)
> >> +  ;; It's somewhat brittle to rely on Dired.
> >> +  (let ((re (dired-glob-regexp ignore)))
> >> +    ;; We could implement root anchoring here, but \\= doesn't work in
> >> +    ;; string-match :-(.
> >> +    (concat (unless (eq ?/ (aref re 3)) "/")
> >> +            ;; Cutting off the anchors.
> >> +            (substring re 2 (- (length re) 2))
> >> +            (unless (eq ?/ (aref re (- (length re) 3)))
> >> +              ;; Either match a full name segment, or eos.
> >> +              "\\(?:/\\|\\'\\)"))))
> > 
> > Same here: what is the purpose of comparisons with a slash?  I think
> > we need some more comments there explaining the logic of the code.
> 
> We compare with a slash to see whether the glob was matching against a 
> directory (in which case it's already anchored to the name of a file 
> name segment), otherwise we add such anchoring to either the end of a 
> file name segment or eos (thus allowing a glob match both directory 
> names and file names).
> 
> Added a shorter comment saying the same.

Thanks, but I miss in that comment explanations of the "magic"
constants 2 and 3.  Could we add that, please?

> >> +(defun etags-regen--append-tags (&rest file-names)
> >> +  (goto-char (point-max))
> >> +  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
> >> +        (inhibit-read-only t))
> >> +    ;; XXX: call-process is significantly faster, though.
> >> +    ;; Like 10ms vs 20ms here.
> >> +    (shell-command
> >> +     (format "%s %s %s -o -"
> >> +             etags-regen-program (mapconcat #'identity options " ")
> >> +             (mapconcat #'identity file-names " "))
> >> +     t etags-regen--errors-buffer-name))
> > 
> > Should we indeed use call-process?
> 
> Something for later improvement.
> 
> Looking at the code, I believe I decided to use 'shell-command' for the 
> first version because of how easy it makes to output stderr to a 
> separate buffer. call-process only offers writing them to a file.

How about mentioning this issue in that comment?

> +(defcustom etags-regen-tags-file "TAGS"
> +  "Name of the tags file to create inside the project.

I suggest

  Name of the tags file to create inside the project by `etags-regen-mode'.

> +(defcustom etags-regen-program-options nil
> +  "List of additional options to pass to the etags program."

I suggest

  List of additional options for etags program invoked by `etags-regen-mode'.

> +(defcustom etags-regen-regexp-alist nil
> +  "Mapping of languages to additional regexps for tags.

I suggest

  Mapping of languages to etags regexps for `etags-regen-mode'.

> +(define-minor-mode etags-regen-mode
> +  "Generate and update the tags automatically.

I suggest

  Minor mode to automatically generate and update tags tables.

Thanks.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 5279 bytes --]

On 30/12/2023 09:33, Eli Zaretskii wrote:
>> Date: Sat, 30 Dec 2023 05:05:01 +0200
>> Cc: 67687@debbugs.gnu.org, eskinjp@gmail.com, stefankangas@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> +(defcustom etags-regen-tags-file "TAGS"
>>>> +  "Name of the tags file to create inside the project.
>>>
>>> This and the other defcustom's here should say in the first line of
>>> the doc string that they are for etags-regen-mode.  This will help
>>> discoverability and also produce a more helpful display with the
>>> various apropos commands.
>>
>> I've tried, but it seems hard to fit into most of them while keeping to
>> the requisite max number of columns. Only managed to fit that into
>> etags-regen-program and etags-regen-file-extensions.
>>
>> TBH, most of the time it would seem superfluous, given the namespaced
>> names. But it's probably good to mention in 'etags-regen-program', on
>> balance.
> 
> I suggest some minor improvements in this area below.

All right.

>>>> +;;;###autoload
>>>> +(put 'etags-regen-file-extensions 'safe-local-variable
>>>> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
>>>
>>> Why not use list-of-strings-p here?
>>
>> Again, that "core ELPA" consideration. We could deploy this feature to a
>> number of released Emacs versions, if we don't introduce such dependencies.
> 
> Isn't this covered by the compat package on ELPA?  If not, I think it
> should be.

These forms go into generated autoloads file for each installed package 
(*-autoloads.el). I think compat doesn't make list-of-string-p 
autoloaded, and autoloads files don't usually have (require ...) forms.

So while I haven't really tested this and could be missing something, it 
seems brittle to rely on 'compat' for this function (if at all possible).

>>>> +     (lambda (f) (or (not (string-match-p match-re f))
>>>> +                (string-match-p "/\\.#" f)
>>>
>>> Is that '/' there to detect regexps for absolute file names?  If so,
>>> that won't work for Windows.
>>
>> It's to detect backup files.
> 
> Can you add a comment there to that effect?

Added.

>>>> +(defun etags-regen--ignore-regexp (ignore)
>>>> +  (require 'dired)
>>>> +  ;; It's somewhat brittle to rely on Dired.
>>>> +  (let ((re (dired-glob-regexp ignore)))
>>>> +    ;; We could implement root anchoring here, but \\= doesn't work in
>>>> +    ;; string-match :-(.
>>>> +    (concat (unless (eq ?/ (aref re 3)) "/")
>>>> +            ;; Cutting off the anchors.
>>>> +            (substring re 2 (- (length re) 2))
>>>> +            (unless (eq ?/ (aref re (- (length re) 3)))
>>>> +              ;; Either match a full name segment, or eos.
>>>> +              "\\(?:/\\|\\'\\)"))))
>>>
>>> Same here: what is the purpose of comparisons with a slash?  I think
>>> we need some more comments there explaining the logic of the code.
>>
>> We compare with a slash to see whether the glob was matching against a
>> directory (in which case it's already anchored to the name of a file
>> name segment), otherwise we add such anchoring to either the end of a
>> file name segment or eos (thus allowing a glob match both directory
>> names and file names).
>>
>> Added a shorter comment saying the same.
> 
> Thanks, but I miss in that comment explanations of the "magic"
> constants 2 and 3.  Could we add that, please?

2 is the length of both anchors, 3 is the index of the character right 
after the anchor. There is already a comment about cutting off the 
anchors, I've expanded it a bit.

>>>> +(defun etags-regen--append-tags (&rest file-names)
>>>> +  (goto-char (point-max))
>>>> +  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
>>>> +        (inhibit-read-only t))
>>>> +    ;; XXX: call-process is significantly faster, though.
>>>> +    ;; Like 10ms vs 20ms here.
>>>> +    (shell-command
>>>> +     (format "%s %s %s -o -"
>>>> +             etags-regen-program (mapconcat #'identity options " ")
>>>> +             (mapconcat #'identity file-names " "))
>>>> +     t etags-regen--errors-buffer-name))
>>>
>>> Should we indeed use call-process?
>>
>> Something for later improvement.
>>
>> Looking at the code, I believe I decided to use 'shell-command' for the
>> first version because of how easy it makes to output stderr to a
>> separate buffer. call-process only offers writing them to a file.
> 
> How about mentioning this issue in that comment?

Added.

>> +(defcustom etags-regen-tags-file "TAGS"
>> +  "Name of the tags file to create inside the project.
> 
> I suggest
> 
>    Name of the tags file to create inside the project by `etags-regen-mode'.
> 
>> +(defcustom etags-regen-program-options nil
>> +  "List of additional options to pass to the etags program."
> 
> I suggest
> 
>    List of additional options for etags program invoked by `etags-regen-mode'.
> 
>> +(defcustom etags-regen-regexp-alist nil
>> +  "Mapping of languages to additional regexps for tags.
> 
> I suggest
> 
>    Mapping of languages to etags regexps for `etags-regen-mode'.
> 
>> +(define-minor-mode etags-regen-mode
>> +  "Generate and update the tags automatically.
> 
> I suggest
> 
>    Minor mode to automatically generate and update tags tables.

Replaced, thanks.

Latest revision attached. Any further comments?

[-- Attachment #2: etags-regen-v5.diff --]
[-- Type: text/x-patch, Size: 18295 bytes --]

diff --git a/.dir-locals.el b/.dir-locals.el
index e087aa89cd1..ce7febca851 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -8,6 +8,12 @@
          (vc-git-annotate-switches . "-w")
          (bug-reference-url-format . "https://debbugs.gnu.org/%s")
 	 (diff-add-log-use-relative-names . t)
+         (etags-regen-regexp-alist
+          .
+          ((("c" "objc") .
+            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
+             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
+         (etags-regen-ignores . ("test/manual/etags/"))
          (vc-prepare-patches-separately . nil)))
  (c-mode . ((c-file-style . "GNU")
             (c-noise-macro-names . ("INLINE" "NO_INLINE" "ATTRIBUTE_NO_SANITIZE_UNDEFINED"
diff --git a/etc/NEWS b/etc/NEWS
index f82564946b7..6d6bca187de 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1243,6 +1243,11 @@ the needs of users with red-green or blue-yellow color deficiency.
 The Info manual "(modus-themes) Top" describes the details and
 showcases all their customization options.
 
+** New global minor mode 'etags-regen-mode'.
+This minor mode generates the tags table automatically based on the
+current project configuration, and later updates it as you edit the
+files and save the changes.
+
 \f
 * Incompatible Lisp Changes in Emacs 30.1
 
diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
new file mode 100644
index 00000000000..e1fca1c4e44
--- /dev/null
+++ b/lisp/progmodes/etags-regen.el
@@ -0,0 +1,424 @@
+;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2021-2023 Free Software Foundation, Inc.
+
+;; Author: Dmitry Gutov <dmitry@gutov.dev>
+;; Keywords: tools
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Simple automatic tags generation with updates on save.
+;;
+;; This mode provides automatic indexing for Emacs "go to definition"
+;; feature, the `xref-go-forward' command (bound to `M-.' by default).
+;;
+;; At the moment reindexing works off before/after-save-hook, but to
+;; handle more complex changes (for example, the user switching to
+;; another branch from the terminal) we can look into plugging into
+;; something like `filenotify'.
+;;
+;; Note that this feature disables itself if the user has some tags
+;; table already visited (with `M-x visit-tags-table', or through an
+;; explicit prompt triggered by some feature that requires tags).
+
+;;; Code:
+
+(require 'cl-lib)
+
+(defgroup etags-regen nil
+  "Auto-(re)generating tags."
+  :group 'tools)
+
+(defvar etags-regen--tags-file nil)
+(defvar etags-regen--tags-root nil)
+(defvar etags-regen--new-file nil)
+
+(declare-function project-root "project")
+(declare-function project-files "project")
+(declare-function dired-glob-regexp "dired")
+
+(defcustom etags-regen-program (executable-find "etags")
+  "Name of the etags program used by `etags-regen-mode'.
+
+If you only have `ctags' installed, you can also set this to
+\"ctags -e\".  Some features might not be supported this way."
+  ;; Always having our 'etags' here would be easier, but we can't
+  ;; always rely on it being installed.  So it might be ctags's etags.
+  :type 'file
+  :version "30.1")
+
+(defcustom etags-regen-tags-file "TAGS"
+  "Name of the tags file to create inside the project by `etags-regen-mode'.
+
+The value should either be a simple file name (no directory
+specified), or a function that accepts the project root directory
+and returns a distinct absolute file name for its tags file.  The
+latter possibility is useful when you prefer to store the tag
+files somewhere else, for example in `temporary-file-directory'."
+  :type '(choice (string :tag "File name")
+                 (function :tag "Function that returns file name"))
+  :version "30.1")
+
+(defcustom etags-regen-program-options nil
+  "List of additional options for etags program invoked by `etags-regen-mode'."
+  :type '(repeat string)
+  :version "30.1")
+
+(defcustom etags-regen-regexp-alist nil
+  "Mapping of languages to etags regexps for `etags-regen-mode'.
+
+These regexps are used in addition to the tags made with the
+standard parsing based on the language.
+
+The value must be a list of conses (LANGUAGES . TAG-REGEXPS)
+where both car and cdr are lists of strings.
+
+Each language should be one of the recognized by etags, see
+`etags --help'.  Each tag regexp should be a string in the format
+as documented for the `--regex' arguments (without `{language}').
+
+We currently support only Emacs's etags program with this option."
+  :type '(repeat
+          (cons
+           :tag "Languages group"
+           (repeat (string :tag "Language name"))
+           (repeat (string :tag "Tag Regexp"))))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-regexp-alist 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p
+             (lambda (group)
+               (and (consp group)
+                    (listp (car group))
+                    (listp (cdr group))
+                    (seq-every-p #'stringp (car group))
+                    (seq-every-p #'stringp (cdr group))))
+             value))))
+
+;; We have to list all extensions: etags falls back to Fortran
+;; when it cannot determine the type of the file.
+;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
+(defcustom etags-regen-file-extensions
+  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
+    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
+    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
+  "Code file extensions for `etags-regen-mode'.
+
+File extensions to generate the tags for."
+  :type '(repeat (string :tag "File extension"))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-file-extensions 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+;; FIXME: We don't support root anchoring yet.
+(defcustom etags-regen-ignores nil
+  "Additional ignore rules, in the format of `project-ignores'."
+  :type '(repeat
+          (string :tag "Glob to ignore"))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-ignores 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
+
+(defvar etags-regen--rescan-files-limit 100)
+
+(defun etags-regen--all-mtimes (proj)
+  (let ((files (etags-regen--all-files proj))
+        (mtimes (make-hash-table :test 'equal))
+        file-name-handler-alist)
+    (dolist (f files)
+      (condition-case nil
+          (puthash f
+                   (file-attribute-modification-time
+                    (file-attributes f))
+                   mtimes)
+        (file-missing nil)))
+    mtimes))
+
+(defun etags-regen--choose-tags-file (proj)
+  (if (functionp etags-regen-tags-file)
+      (funcall etags-regen-tags-file (project-root proj))
+    (expand-file-name etags-regen-tags-file (project-root proj))))
+
+(defun etags-regen--refresh (proj)
+  (save-excursion
+    (let* ((tags-file (etags-regen--choose-tags-file proj))
+           (tags-mtime (file-attribute-modification-time
+                        (file-attributes tags-file)))
+           (all-mtimes (etags-regen--all-mtimes proj))
+           added-files
+           changed-files
+           removed-files)
+      (etags-regen--visit-table tags-file (project-root proj))
+      (set-buffer (get-file-buffer tags-file))
+      (dolist (file (tags-table-files))
+        (let ((mtime (gethash file all-mtimes)))
+          (cond
+           ((null mtime)
+            (push file removed-files))
+           ((time-less-p tags-mtime mtime)
+            (push file changed-files)
+            (remhash file all-mtimes))
+           (t
+            (remhash file all-mtimes)))))
+      (maphash
+       (lambda (key _value)
+         (push key added-files))
+       all-mtimes)
+      (if (> (+ (length added-files)
+                (length changed-files)
+                (length removed-files))
+             etags-regen--rescan-files-limit)
+          (progn
+            (message "etags-regen: Too many changes, falling back to full rescan")
+            (etags-regen--tags-cleanup))
+        (dolist (file (nconc removed-files changed-files))
+          (etags-regen--remove-tag file))
+        (when (or changed-files added-files)
+          (apply #'etags-regen--append-tags
+                 (nconc changed-files added-files)))
+        (when (or changed-files added-files removed-files)
+          (let ((save-silently t)
+                (message-log-max nil))
+            (save-buffer 0)))))))
+
+(defun etags-regen--maybe-generate ()
+  (let (proj)
+    (when (and etags-regen--tags-root
+               (not (file-in-directory-p default-directory
+                                         etags-regen--tags-root)))
+      (etags-regen--tags-cleanup))
+    (when (and (not etags-regen--tags-root)
+               ;; If existing table is visited that's not generated by
+               ;; this mode, skip all functionality.
+               (not (or tags-file-name
+                        tags-table-list))
+               (file-exists-p (etags-regen--choose-tags-file
+                               (setq proj (project-current)))))
+      (message "Found existing tags table, refreshing...")
+      (etags-regen--refresh proj))
+    (when (and (not (or tags-file-name
+                        tags-table-list))
+               (setq proj (or proj (project-current))))
+      (message "Generating new tags table...")
+      (let ((start (time-to-seconds)))
+        (etags-regen--tags-generate proj)
+        (message "...done (%.2f s)" (- (time-to-seconds) start))))))
+
+(defun etags-regen--all-files (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         ;; TODO: Make the scanning more efficient, e.g. move the
+         ;; filtering by glob to project (project-files-filtered...).
+         (files (project-files proj))
+         (match-re (concat
+                    "\\."
+                    (regexp-opt etags-regen-file-extensions)
+                    "\\'"))
+         (ir-start (1- (length root)))
+         (ignores-regexps
+          (mapcar #'etags-regen--ignore-regexp
+                  etags-regen-ignores)))
+    (cl-delete-if
+     (lambda (f) (or (not (string-match-p match-re f))
+                (string-match-p "/\\.#" f) ;Backup files.
+                (cl-some (lambda (ignore) (string-match ignore f ir-start))
+                         ignores-regexps)))
+     files)))
+
+(defun etags-regen--ignore-regexp (ignore)
+  (require 'dired)
+  ;; It's somewhat brittle to rely on Dired.
+  (let ((re (dired-glob-regexp ignore)))
+    ;; We could implement root anchoring here, but \\= doesn't work in
+    ;; string-match :-(.
+    (concat (unless (eq ?/ (aref re 3)) "/")
+            ;; Cutting off the anchors added by `dired-glob-regexp'.
+            (substring re 2 (- (length re) 2))
+            ;; This way we allow a glob to match against a directory
+            ;; name, or a file name.  And when it ends with / already,
+            ;; no need to add the anchoring.
+            (unless (eq ?/ (aref re (- (length re) 3)))
+              ;; Either match a full name segment, or eos.
+              "\\(?:/\\|\\'\\)"))))
+
+(defun etags-regen--tags-generate (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         (files (etags-regen--all-files proj))
+         (tags-file (etags-regen--choose-tags-file proj))
+         (ctags-p (etags-regen--ctags-p))
+         (command (format "%s %s %s - -o %s"
+                          etags-regen-program
+                          (mapconcat #'identity
+                                     (etags-regen--build-program-options ctags-p)
+                                     " ")
+                          ;; ctags's etags requires '-L' for stdin input.
+                          (if ctags-p "-L" "")
+                          tags-file)))
+    (with-temp-buffer
+      (mapc (lambda (f)
+              (insert f "\n"))
+            files)
+      (shell-command-on-region (point-min) (point-max) command
+                               nil nil etags-regen--errors-buffer-name t))
+    (etags-regen--visit-table tags-file root)))
+
+(defun etags-regen--visit-table (tags-file root)
+  ;; Invalidate the scanned tags after any change is written to disk.
+  (add-hook 'after-save-hook #'etags-regen--update-file)
+  (add-hook 'before-save-hook #'etags-regen--mark-as-new)
+  (setq etags-regen--tags-file tags-file
+        etags-regen--tags-root root)
+  (visit-tags-table etags-regen--tags-file))
+
+(defun etags-regen--ctags-p ()
+  (string-search "Ctags"
+                 (shell-command-to-string
+                  (format "%s --version" etags-regen-program))))
+
+(defun etags-regen--build-program-options (ctags-p)
+  (when (and etags-regen-regexp-alist ctags-p)
+    (user-error "etags-regen-regexp-alist is not supported with Ctags"))
+  (nconc
+   (mapcan
+    (lambda (group)
+      (mapcan
+       (lambda (lang)
+         (mapcar (lambda (regexp)
+                   (concat "--regex="
+                           (shell-quote-argument
+                            (format "{%s}%s" lang regexp))))
+                 (cdr group)))
+       (car group)))
+    etags-regen-regexp-alist)
+   etags-regen-program-options))
+
+(defun etags-regen--update-file ()
+  ;; TODO: Maybe only do this when Emacs is idle for a bit.  Or defer
+  ;; the updates and do them later in bursts when the table is used.
+  (let* ((file-name buffer-file-name)
+         (tags-file-buf (and etags-regen--tags-root
+                             (get-file-buffer etags-regen--tags-file)))
+         (relname (concat "/" (file-relative-name file-name
+                                                  etags-regen--tags-root)))
+         (ignores etags-regen-ignores)
+         pr should-scan)
+    (save-excursion
+      (when tags-file-buf
+        (cond
+         ((and etags-regen--new-file
+               (kill-local-variable 'etags-regen--new-file)
+               (setq pr (project-current))
+               (equal (project-root pr) etags-regen--tags-root)
+               (member file-name (project-files pr)))
+          (set-buffer tags-file-buf)
+          (setq should-scan t))
+         ((progn (set-buffer tags-file-buf)
+                 (etags-regen--remove-tag file-name))
+          (setq should-scan t))))
+      (when (and should-scan
+                 (not (cl-some
+                       (lambda (ignore)
+                         (string-match-p
+                          (etags-regen--ignore-regexp ignore)
+                          relname))
+                       ignores)))
+        (etags-regen--append-tags file-name)
+        (let ((save-silently t)
+              (message-log-max nil))
+          (save-buffer 0))))))
+
+(defun etags-regen--remove-tag (file-name)
+  (goto-char (point-min))
+  (when (search-forward (format "\f\n%s," file-name) nil t)
+    (let ((start (match-beginning 0)))
+      (search-forward "\f\n" nil 'move)
+      (let ((inhibit-read-only t))
+        (delete-region start
+                       (if (eobp)
+                           (point)
+                         (- (point) 2)))))
+    t))
+
+(defun etags-regen--append-tags (&rest file-names)
+  (goto-char (point-max))
+  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
+        (inhibit-read-only t))
+    ;; XXX: call-process is significantly faster, though.
+    ;; Like 10ms vs 20ms here.  But `shell-command' makes it easy to
+    ;; direct stderr to a separate buffer.
+    (shell-command
+     (format "%s %s %s -o -"
+             etags-regen-program (mapconcat #'identity options " ")
+             (mapconcat #'identity file-names " "))
+     t etags-regen--errors-buffer-name))
+  ;; FIXME: Is there a better way to do this?
+  ;; Completion table is the only remaining place where the
+  ;; update is not incremental.
+  (setq-default tags-completion-table nil))
+
+(defun etags-regen--mark-as-new ()
+  (when (and etags-regen--tags-root
+             (not buffer-file-number))
+    (setq-local etags-regen--new-file t)))
+
+(defun etags-regen--tags-cleanup ()
+  (when etags-regen--tags-file
+    (let ((buffer (get-file-buffer etags-regen--tags-file)))
+      (and buffer
+           (kill-buffer buffer)))
+    (tags-reset-tags-tables)
+    (setq tags-file-name nil
+          tags-table-list nil
+          etags-regen--tags-file nil
+          etags-regen--tags-root nil))
+  (remove-hook 'after-save-hook #'etags-regen--update-file)
+  (remove-hook 'before-save-hook #'etags-regen--mark-as-new))
+
+(defvar etags-regen-mode-map (make-sparse-keymap))
+
+;;;###autoload
+(define-minor-mode etags-regen-mode
+  "Minor mode to automatically generate and update tags tables.
+
+This minor mode generates the tags table automatically based on
+the current project configuration, and later updates it as you
+edit the files and save the changes."
+  :global t
+  (if etags-regen-mode
+      (progn
+        (advice-add 'etags--xref-backend :before
+                    #'etags-regen--maybe-generate)
+        (advice-add 'tags-completion-at-point-function :before
+                    #'etags-regen--maybe-generate))
+    (advice-remove 'etags--xref-backend #'etags-regen--maybe-generate)
+    (advice-remove 'tags-completion-at-point-function #'etags-regen--maybe-generate)
+    (etags-regen--tags-cleanup)))
+
+(provide 'etags-regen)
+
+;;; etags-regen.el ends here

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

My review below.  I wasn't paying attention to the full discussion, so
please just ignore any points that you have already discussed with Eli.

Dmitry Gutov <dmitry@gutov.dev> writes:

> diff --git a/etc/NEWS b/etc/NEWS
> index f82564946b7..6d6bca187de 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -1243,6 +1243,11 @@ the needs of users with red-green or blue-yellow color deficiency.
>  The Info manual "(modus-themes) Top" describes the details and
>  showcases all their customization options.
>
> +** New global minor mode 'etags-regen-mode'.
> +This minor mode generates the tags table automatically based on the
> +current project configuration, and later updates it as you edit the
> +files and save the changes.

Consider referring to the relevant section in the info manual.

> +
>  \f
>  * Incompatible Lisp Changes in Emacs 30.1
>
> diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
> new file mode 100644
> index 00000000000..e1fca1c4e44
> --- /dev/null
> +++ b/lisp/progmodes/etags-regen.el
> @@ -0,0 +1,424 @@
> +;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
> +
> +;; Copyright (C) 2021-2023 Free Software Foundation, Inc.
> +
> +;; Author: Dmitry Gutov <dmitry@gutov.dev>
> +;; Keywords: tools
> +
> +;; This file is part of GNU Emacs.
> +
> +;; GNU Emacs is free software: you can redistribute it and/or modify
> +;; it under the terms of the GNU General Public License as published by
> +;; the Free Software Foundation, either version 3 of the License, or
> +;; (at your option) any later version.
> +
> +;; GNU Emacs is distributed in the hope that it will be useful,
> +;; but WITHOUT ANY WARRANTY; without even the implied warranty of
> +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;; GNU General Public License for more details.
> +
> +;; You should have received a copy of the GNU General Public License
> +;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
> +
> +;;; Commentary:
> +
> +;; Simple automatic tags generation with updates on save.
> +;;
> +;; This mode provides automatic indexing for Emacs "go to definition"
> +;; feature, the `xref-go-forward' command (bound to `M-.' by default).
> +;;
> +;; At the moment reindexing works off before/after-save-hook, but to
> +;; handle more complex changes (for example, the user switching to
> +;; another branch from the terminal) we can look into plugging into
> +;; something like `filenotify'.
> +;;
> +;; Note that this feature disables itself if the user has some tags
> +;; table already visited (with `M-x visit-tags-table', or through an
> +;; explicit prompt triggered by some feature that requires tags).
> +
> +;;; Code:
> +
> +(require 'cl-lib)
> +
> +(defgroup etags-regen nil
> +  "Auto-(re)generating tags."
> +  :group 'tools)

How about:

    "Auto-generate \"tags\" file."

> +
> +(defvar etags-regen--tags-file nil)
> +(defvar etags-regen--tags-root nil)
> +(defvar etags-regen--new-file nil)
> +
> +(declare-function project-root "project")
> +(declare-function project-files "project")
> +(declare-function dired-glob-regexp "dired")
> +
> +(defcustom etags-regen-program (executable-find "etags")
> +  "Name of the etags program used by `etags-regen-mode'.
> +
> +If you only have `ctags' installed, you can also set this to
> +\"ctags -e\".  Some features might not be supported this way."
> +  ;; Always having our 'etags' here would be easier, but we can't
> +  ;; always rely on it being installed.  So it might be ctags's etags.
> +  :type 'file
> +  :version "30.1")
> +
> +(defcustom etags-regen-tags-file "TAGS"
> +  "Name of the tags file to create inside the project by `etags-regen-mode'.
> +
> +The value should either be a simple file name (no directory
> +specified), or a function that accepts the project root directory
> +and returns a distinct absolute file name for its tags file.  The
> +latter possibility is useful when you prefer to store the tag
> +files somewhere else, for example in `temporary-file-directory'."
> +  :type '(choice (string :tag "File name")
> +                 (function :tag "Function that returns file name"))
> +  :version "30.1")

Did you consider making it always store the TAGS files somewhere in
`temporary-file-directory'?  They should be rather ephemereal in any
case, I think?

In that case, we could perhaps even ignore an existing TAGS file, if
this mode is enabled.  Perhaps as an option.

> +
> +(defcustom etags-regen-program-options nil
> +  "List of additional options for etags program invoked by `etags-regen-mode'."
> +  :type '(repeat string)
> +  :version "30.1")

Perhaps add:

    See the full list of options that `etags' accepts in Info node
    `(emacs) Create Tags Table'.

Should this be marked unsafe?  Actually, same question for all of the
above defcustoms, given that we use `shell-command-to-string'.

Speaking of which, should we have more `shell-quote-argument' below?
I didn't look at every example, but maybe you did.

> +
> +(defcustom etags-regen-regexp-alist nil
> +  "Mapping of languages to etags regexps for `etags-regen-mode'.
> +
> +These regexps are used in addition to the tags made with the
> +standard parsing based on the language.
> +
> +The value must be a list of conses (LANGUAGES . TAG-REGEXPS)
> +where both car and cdr are lists of strings.

I think that should better be:

    where both LANGUAGES and TAG-REGEXPS are lists of strings.

I'm not sure we should say "conses" there, or should we?  I think we
usually prefer something like:

    The value is a list where each element has the form
    (LANGUAGES . TAG-REGEXPS)

I think this is better because it sounds less foreign to random users
with no background in ELisp.

But Eli is much better than me at this.  :-)

> +
> +Each language should be one of the recognized by etags, see
> +`etags --help'.  Each tag regexp should be a string in the format
> +as documented for the `--regex' arguments (without `{language}').
   ^^

Nit, but I think "as" could be scratched.

> +
> +We currently support only Emacs's etags program with this option."
> +  :type '(repeat
> +          (cons
> +           :tag "Languages group"
> +           (repeat (string :tag "Language name"))
> +           (repeat (string :tag "Tag Regexp"))))
> +  :version "30.1")
> +
> +;;;###autoload
> +(put 'etags-regen-regexp-alist 'safe-local-variable
> +     (lambda (value)
> +       (and (listp value)
> +            (seq-every-p
> +             (lambda (group)
> +               (and (consp group)
> +                    (listp (car group))
> +                    (listp (cdr group))
> +                    (seq-every-p #'stringp (car group))
> +                    (seq-every-p #'stringp (cdr group))))
> +             value))))
> +
> +;; We have to list all extensions: etags falls back to Fortran
> +;; when it cannot determine the type of the file.

(A battle-tested default, if nothing else.  ;-)

> +;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
> +(defcustom etags-regen-file-extensions
> +  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
> +    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
> +    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
> +  "Code file extensions for `etags-regen-mode'.
> +File extensions to generate the tags for."
> +  :type '(repeat (string :tag "File extension"))
> +  :version "30.1")
> +
> +;;;###autoload
> +(put 'etags-regen-file-extensions 'safe-local-variable
> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
> +
> +;; FIXME: We don't support root anchoring yet.

What is root anchoring?  Does it deserve a sentence that explains what
it is?

> +(defcustom etags-regen-ignores nil
> +  "Additional ignore rules, in the format of `project-ignores'."
> +  :type '(repeat
> +          (string :tag "Glob to ignore"))
> +  :version "30.1")
> +
> +;;;###autoload
> +(put 'etags-regen-ignores 'safe-local-variable
> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
> +
> +(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
> +
> +(defvar etags-regen--rescan-files-limit 100)
> +
> +(defun etags-regen--all-mtimes (proj)
> +  (let ((files (etags-regen--all-files proj))
> +        (mtimes (make-hash-table :test 'equal))
> +        file-name-handler-alist)
> +    (dolist (f files)
> +      (condition-case nil
> +          (puthash f
> +                   (file-attribute-modification-time
> +                    (file-attributes f))
> +                   mtimes)
> +        (file-missing nil)))
> +    mtimes))

Could we use file notifications for this?  Maybe as a future
improvement.

Other than that, LGTM.

Thanks again for working on this.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 7739 bytes --]

On 31/12/2023 03:02, Stefan Kangas wrote:
> My review below.  I wasn't paying attention to the full discussion, so
> please just ignore any points that you have already discussed with Eli.

Thanks.

> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> diff --git a/etc/NEWS b/etc/NEWS
>> index f82564946b7..6d6bca187de 100644
>> --- a/etc/NEWS
>> +++ b/etc/NEWS
>> @@ -1243,6 +1243,11 @@ the needs of users with red-green or blue-yellow color deficiency.
>>   The Info manual "(modus-themes) Top" describes the details and
>>   showcases all their customization options.
>>
>> +** New global minor mode 'etags-regen-mode'.
>> +This minor mode generates the tags table automatically based on the
>> +current project configuration, and later updates it as you edit the
>> +files and save the changes.
> 
> Consider referring to the relevant section in the info manual.

I guess we'll add that reference later when/if we document it in the manual.

>> +(require 'cl-lib)
>> +
>> +(defgroup etags-regen nil
>> +  "Auto-(re)generating tags."
>> +  :group 'tools)
> 
> How about:
> 
>      "Auto-generate \"tags\" file."

Sorry, I don't understand the meaning of the quotes. The file name is 
TAGS, if we wanted to refer to it. You also dropped the bit in quotes 
that implies the automatic updates, too.

We can say "Auto-(re)generate tags file". Though knowing how the tags 
are stored might not be that necessary, if the generation and 
loading/refresh happens automatically.

>> +(defcustom etags-regen-tags-file "TAGS"
>> +  "Name of the tags file to create inside the project by `etags-regen-mode'.
>> +
>> +The value should either be a simple file name (no directory
>> +specified), or a function that accepts the project root directory
>> +and returns a distinct absolute file name for its tags file.  The
>> +latter possibility is useful when you prefer to store the tag
>> +files somewhere else, for example in `temporary-file-directory'."
>> +  :type '(choice (string :tag "File name")
>> +                 (function :tag "Function that returns file name"))
>> +  :version "30.1")
> 
> Did you consider making it always store the TAGS files somewhere in
> `temporary-file-directory'?  They should be rather ephemereal in any
> case, I think?

This was my approach originally, but the downside was always having to 
generate the tags fully when Emacs is restarted. The current code also 
refreshes existing tags file when there are not too many changed files 
to process.

Now that etags-regen-tags-file can be a function that returns stable 
file names corresponding to project root even in /tmp, that feature can 
still work (until /tmp is cleared).

Whether that should be the default, I don't know. Perhaps people will 
like the easier access to the generated file that the current default 
allows.

> In that case, we could perhaps even ignore an existing TAGS file, if
> this mode is enabled.  Perhaps as an option.

This will happen if no tags file is visited and (funcall 
etags-regen-tags-file root) returns a different file.

>> +
>> +(defcustom etags-regen-program-options nil
>> +  "List of additional options for etags program invoked by `etags-regen-mode'."
>> +  :type '(repeat string)
>> +  :version "30.1")
> 
> Perhaps add:
> 
>      See the full list of options that `etags' accepts in Info node
>      `(emacs) Create Tags Table'.
> 
> Should this be marked unsafe?  Actually, same question for all of the
> above defcustoms, given that we use `shell-command-to-string'.

Given that this option doesn't have the 'safe-local-variable' property, 
do we need to do something else?

> Speaking of which, should we have more `shell-quote-argument' below?
> I didn't look at every example, but maybe you did.

Indeed, the processing of etags-regen-program-options was missing the 
conversion through shell-quote-argument. In case, for example. someone 
adds an option with a value that includes a space.

>> +
>> +(defcustom etags-regen-regexp-alist nil
>> +  "Mapping of languages to etags regexps for `etags-regen-mode'.
>> +
>> +These regexps are used in addition to the tags made with the
>> +standard parsing based on the language.
>> +
>> +The value must be a list of conses (LANGUAGES . TAG-REGEXPS)
>> +where both car and cdr are lists of strings.
> 
> I think that should better be:
> 
>      where both LANGUAGES and TAG-REGEXPS are lists of strings.
> 
> I'm not sure we should say "conses" there, or should we?  I think we
> usually prefer something like:
> 
>      The value is a list where each element has the form
>      (LANGUAGES . TAG-REGEXPS)
> 
> I think this is better because it sounds less foreign to random users
> with no background in ELisp.
> 
> But Eli is much better than me at this.  :-)

Why not, I've replaced the text with your versions.

>> +
>> +Each language should be one of the recognized by etags, see
>> +`etags --help'.  Each tag regexp should be a string in the format
>> +as documented for the `--regex' arguments (without `{language}').
>     ^^
> 
> Nit, but I think "as" could be scratched.

Ok.

>> +
>> +We currently support only Emacs's etags program with this option."
>> +  :type '(repeat
>> +          (cons
>> +           :tag "Languages group"
>> +           (repeat (string :tag "Language name"))
>> +           (repeat (string :tag "Tag Regexp"))))
>> +  :version "30.1")
>> +
>> +;;;###autoload
>> +(put 'etags-regen-regexp-alist 'safe-local-variable
>> +     (lambda (value)
>> +       (and (listp value)
>> +            (seq-every-p
>> +             (lambda (group)
>> +               (and (consp group)
>> +                    (listp (car group))
>> +                    (listp (cdr group))
>> +                    (seq-every-p #'stringp (car group))
>> +                    (seq-every-p #'stringp (cdr group))))
>> +             value))))
>> +
>> +;; We have to list all extensions: etags falls back to Fortran
>> +;; when it cannot determine the type of the file.
> 
> (A battle-tested default, if nothing else.  ;-)

I'd rather etags skipped unknown files, but there is no such option.

>> +;;;###autoload
>> +(put 'etags-regen-file-extensions 'safe-local-variable
>> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
>> +
>> +;; FIXME: We don't support root anchoring yet.
> 
> What is root anchoring?  Does it deserve a sentence that explains what
> it is?

It's the "to root an entry" thing described in the docstring for 
`project-ignores', which is referenced.

>> +(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
>> +
>> +(defvar etags-regen--rescan-files-limit 100)
>> +
>> +(defun etags-regen--all-mtimes (proj)
>> +  (let ((files (etags-regen--all-files proj))
>> +        (mtimes (make-hash-table :test 'equal))
>> +        file-name-handler-alist)
>> +    (dolist (f files)
>> +      (condition-case nil
>> +          (puthash f
>> +                   (file-attribute-modification-time
>> +                    (file-attributes f))
>> +                   mtimes)
>> +        (file-missing nil)))
>> +    mtimes))
> 
> Could we use file notifications for this?  Maybe as a future
> improvement.

File notifications are mentioned in the Commentary. Someday.

The thread about expected problems with our filenotify integration was 
sometime between these two, but I can't find it now :-(

https://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00369.html

https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg00680.html
https://lists.gnu.org/archive/html/emacs-devel/2021-01/msg00428.html

Perhaps filenotify-recursively will help 
(https://lists.gnu.org/archive/html/emacs-devel/2021-08/msg00876.html).

> Other than that, LGTM.
> 
> Thanks again for working on this.

Thanks!

Next revision attached.

[-- Attachment #2: etags-regen-v6.diff --]
[-- Type: text/x-patch, Size: 18373 bytes --]

diff --git a/.dir-locals.el b/.dir-locals.el
index e087aa89cd1..ce7febca851 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -8,6 +8,12 @@
          (vc-git-annotate-switches . "-w")
          (bug-reference-url-format . "https://debbugs.gnu.org/%s")
 	 (diff-add-log-use-relative-names . t)
+         (etags-regen-regexp-alist
+          .
+          ((("c" "objc") .
+            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
+             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
+         (etags-regen-ignores . ("test/manual/etags/"))
          (vc-prepare-patches-separately . nil)))
  (c-mode . ((c-file-style . "GNU")
             (c-noise-macro-names . ("INLINE" "NO_INLINE" "ATTRIBUTE_NO_SANITIZE_UNDEFINED"
diff --git a/etc/NEWS b/etc/NEWS
index ba40efa2e9d..0928ae5a2dc 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1270,6 +1270,11 @@ the needs of users with red-green or blue-yellow color deficiency.
 The Info manual "(modus-themes) Top" describes the details and
 showcases all their customization options.
 
+** New global minor mode 'etags-regen-mode'.
+This minor mode generates the tags table automatically based on the
+current project configuration, and later updates it as you edit the
+files and save the changes.
+
 \f
 * Incompatible Lisp Changes in Emacs 30.1
 
diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
new file mode 100644
index 00000000000..0c4397abb56
--- /dev/null
+++ b/lisp/progmodes/etags-regen.el
@@ -0,0 +1,426 @@
+;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2021-2023 Free Software Foundation, Inc.
+
+;; Author: Dmitry Gutov <dmitry@gutov.dev>
+;; Keywords: tools
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Simple automatic tags generation with updates on save.
+;;
+;; This mode provides automatic indexing for Emacs "go to definition"
+;; feature, the `xref-go-forward' command (bound to `M-.' by default).
+;;
+;; At the moment reindexing works off before/after-save-hook, but to
+;; handle more complex changes (for example, the user switching to
+;; another branch from the terminal) we can look into plugging into
+;; something like `filenotify'.
+;;
+;; Note that this feature disables itself if the user has some tags
+;; table already visited (with `M-x visit-tags-table', or through an
+;; explicit prompt triggered by some feature that requires tags).
+
+;;; Code:
+
+(require 'cl-lib)
+
+(defgroup etags-regen nil
+  "Auto-(re)generating tags."
+  :group 'tools)
+
+(defvar etags-regen--tags-file nil)
+(defvar etags-regen--tags-root nil)
+(defvar etags-regen--new-file nil)
+
+(declare-function project-root "project")
+(declare-function project-files "project")
+(declare-function dired-glob-regexp "dired")
+
+(defcustom etags-regen-program (executable-find "etags")
+  "Name of the etags program used by `etags-regen-mode'.
+
+If you only have `ctags' installed, you can also set this to
+\"ctags -e\".  Some features might not be supported this way."
+  ;; Always having our 'etags' here would be easier, but we can't
+  ;; always rely on it being installed.  So it might be ctags's etags.
+  :type 'file
+  :version "30.1")
+
+(defcustom etags-regen-tags-file "TAGS"
+  "Name of the tags file to create inside the project by `etags-regen-mode'.
+
+The value should either be a simple file name (no directory
+specified), or a function that accepts the project root directory
+and returns a distinct absolute file name for its tags file.  The
+latter possibility is useful when you prefer to store the tag
+files somewhere else, for example in `temporary-file-directory'."
+  :type '(choice (string :tag "File name")
+                 (function :tag "Function that returns file name"))
+  :version "30.1")
+
+(defcustom etags-regen-program-options nil
+  "List of additional options for etags program invoked by `etags-regen-mode'."
+  :type '(repeat string)
+  :version "30.1")
+
+(defcustom etags-regen-regexp-alist nil
+  "Mapping of languages to etags regexps for `etags-regen-mode'.
+
+These regexps are used in addition to the tags made with the
+standard parsing based on the language.
+
+The value must be a list where each element has the
+form (LANGUAGES . TAG-REGEXPS) where both LANGUAGES and
+TAG-REGEXPS are lists of strings.
+
+Each language should be one of the recognized by etags, see
+`etags --help'.  Each tag regexp should be a string in the format
+documented for the `--regex' arguments (without `{language}').
+
+We currently support only Emacs's etags program with this option."
+  :type '(repeat
+          (cons
+           :tag "Languages group"
+           (repeat (string :tag "Language name"))
+           (repeat (string :tag "Tag Regexp"))))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-regexp-alist 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p
+             (lambda (group)
+               (and (consp group)
+                    (listp (car group))
+                    (listp (cdr group))
+                    (seq-every-p #'stringp (car group))
+                    (seq-every-p #'stringp (cdr group))))
+             value))))
+
+;; We have to list all extensions: etags falls back to Fortran
+;; when it cannot determine the type of the file.
+;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
+(defcustom etags-regen-file-extensions
+  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
+    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
+    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
+  "Code file extensions for `etags-regen-mode'.
+
+File extensions to generate the tags for."
+  :type '(repeat (string :tag "File extension"))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-file-extensions 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+;; FIXME: We don't support root anchoring yet.
+(defcustom etags-regen-ignores nil
+  "Additional ignore rules, in the format of `project-ignores'."
+  :type '(repeat
+          (string :tag "Glob to ignore"))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-ignores 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
+
+(defvar etags-regen--rescan-files-limit 100)
+
+(defun etags-regen--all-mtimes (proj)
+  (let ((files (etags-regen--all-files proj))
+        (mtimes (make-hash-table :test 'equal))
+        file-name-handler-alist)
+    (dolist (f files)
+      (condition-case nil
+          (puthash f
+                   (file-attribute-modification-time
+                    (file-attributes f))
+                   mtimes)
+        (file-missing nil)))
+    mtimes))
+
+(defun etags-regen--choose-tags-file (proj)
+  (if (functionp etags-regen-tags-file)
+      (funcall etags-regen-tags-file (project-root proj))
+    (expand-file-name etags-regen-tags-file (project-root proj))))
+
+(defun etags-regen--refresh (proj)
+  (save-excursion
+    (let* ((tags-file (etags-regen--choose-tags-file proj))
+           (tags-mtime (file-attribute-modification-time
+                        (file-attributes tags-file)))
+           (all-mtimes (etags-regen--all-mtimes proj))
+           added-files
+           changed-files
+           removed-files)
+      (etags-regen--visit-table tags-file (project-root proj))
+      (set-buffer (get-file-buffer tags-file))
+      (dolist (file (tags-table-files))
+        (let ((mtime (gethash file all-mtimes)))
+          (cond
+           ((null mtime)
+            (push file removed-files))
+           ((time-less-p tags-mtime mtime)
+            (push file changed-files)
+            (remhash file all-mtimes))
+           (t
+            (remhash file all-mtimes)))))
+      (maphash
+       (lambda (key _value)
+         (push key added-files))
+       all-mtimes)
+      (if (> (+ (length added-files)
+                (length changed-files)
+                (length removed-files))
+             etags-regen--rescan-files-limit)
+          (progn
+            (message "etags-regen: Too many changes, falling back to full rescan")
+            (etags-regen--tags-cleanup))
+        (dolist (file (nconc removed-files changed-files))
+          (etags-regen--remove-tag file))
+        (when (or changed-files added-files)
+          (apply #'etags-regen--append-tags
+                 (nconc changed-files added-files)))
+        (when (or changed-files added-files removed-files)
+          (let ((save-silently t)
+                (message-log-max nil))
+            (save-buffer 0)))))))
+
+(defun etags-regen--maybe-generate ()
+  (let (proj)
+    (when (and etags-regen--tags-root
+               (not (file-in-directory-p default-directory
+                                         etags-regen--tags-root)))
+      (etags-regen--tags-cleanup))
+    (when (and (not etags-regen--tags-root)
+               ;; If existing table is visited that's not generated by
+               ;; this mode, skip all functionality.
+               (not (or tags-file-name
+                        tags-table-list))
+               (file-exists-p (etags-regen--choose-tags-file
+                               (setq proj (project-current)))))
+      (message "Found existing tags table, refreshing...")
+      (etags-regen--refresh proj))
+    (when (and (not (or tags-file-name
+                        tags-table-list))
+               (setq proj (or proj (project-current))))
+      (message "Generating new tags table...")
+      (let ((start (time-to-seconds)))
+        (etags-regen--tags-generate proj)
+        (message "...done (%.2f s)" (- (time-to-seconds) start))))))
+
+(defun etags-regen--all-files (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         ;; TODO: Make the scanning more efficient, e.g. move the
+         ;; filtering by glob to project (project-files-filtered...).
+         (files (project-files proj))
+         (match-re (concat
+                    "\\."
+                    (regexp-opt etags-regen-file-extensions)
+                    "\\'"))
+         (ir-start (1- (length root)))
+         (ignores-regexps
+          (mapcar #'etags-regen--ignore-regexp
+                  etags-regen-ignores)))
+    (cl-delete-if
+     (lambda (f) (or (not (string-match-p match-re f))
+                (string-match-p "/\\.#" f) ;Backup files.
+                (cl-some (lambda (ignore) (string-match ignore f ir-start))
+                         ignores-regexps)))
+     files)))
+
+(defun etags-regen--ignore-regexp (ignore)
+  (require 'dired)
+  ;; It's somewhat brittle to rely on Dired.
+  (let ((re (dired-glob-regexp ignore)))
+    ;; We could implement root anchoring here, but \\= doesn't work in
+    ;; string-match :-(.
+    (concat (unless (eq ?/ (aref re 3)) "/")
+            ;; Cutting off the anchors added by `dired-glob-regexp'.
+            (substring re 2 (- (length re) 2))
+            ;; This way we allow a glob to match against a directory
+            ;; name, or a file name.  And when it ends with / already,
+            ;; no need to add the anchoring.
+            (unless (eq ?/ (aref re (- (length re) 3)))
+              ;; Either match a full name segment, or eos.
+              "\\(?:/\\|\\'\\)"))))
+
+(defun etags-regen--tags-generate (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         (files (etags-regen--all-files proj))
+         (tags-file (etags-regen--choose-tags-file proj))
+         (ctags-p (etags-regen--ctags-p))
+         (command (format "%s %s %s - -o %s"
+                          etags-regen-program
+                          (mapconcat #'identity
+                                     (etags-regen--build-program-options ctags-p)
+                                     " ")
+                          ;; ctags's etags requires '-L' for stdin input.
+                          (if ctags-p "-L" "")
+                          tags-file)))
+    (with-temp-buffer
+      (mapc (lambda (f)
+              (insert f "\n"))
+            files)
+      (shell-command-on-region (point-min) (point-max) command
+                               nil nil etags-regen--errors-buffer-name t))
+    (etags-regen--visit-table tags-file root)))
+
+(defun etags-regen--visit-table (tags-file root)
+  ;; Invalidate the scanned tags after any change is written to disk.
+  (add-hook 'after-save-hook #'etags-regen--update-file)
+  (add-hook 'before-save-hook #'etags-regen--mark-as-new)
+  (setq etags-regen--tags-file tags-file
+        etags-regen--tags-root root)
+  (visit-tags-table etags-regen--tags-file))
+
+(defun etags-regen--ctags-p ()
+  (string-search "Ctags"
+                 (shell-command-to-string
+                  (format "%s --version" etags-regen-program))))
+
+(defun etags-regen--build-program-options (ctags-p)
+  (when (and etags-regen-regexp-alist ctags-p)
+    (user-error "etags-regen-regexp-alist is not supported with Ctags"))
+  (nconc
+   (mapcan
+    (lambda (group)
+      (mapcan
+       (lambda (lang)
+         (mapcar (lambda (regexp)
+                   (concat "--regex="
+                           (shell-quote-argument
+                            (format "{%s}%s" lang regexp))))
+                 (cdr group)))
+       (car group)))
+    etags-regen-regexp-alist)
+   (mapcar #'shell-quote-argument
+           etags-regen-program-options)))
+
+(defun etags-regen--update-file ()
+  ;; TODO: Maybe only do this when Emacs is idle for a bit.  Or defer
+  ;; the updates and do them later in bursts when the table is used.
+  (let* ((file-name buffer-file-name)
+         (tags-file-buf (and etags-regen--tags-root
+                             (get-file-buffer etags-regen--tags-file)))
+         (relname (concat "/" (file-relative-name file-name
+                                                  etags-regen--tags-root)))
+         (ignores etags-regen-ignores)
+         pr should-scan)
+    (save-excursion
+      (when tags-file-buf
+        (cond
+         ((and etags-regen--new-file
+               (kill-local-variable 'etags-regen--new-file)
+               (setq pr (project-current))
+               (equal (project-root pr) etags-regen--tags-root)
+               (member file-name (project-files pr)))
+          (set-buffer tags-file-buf)
+          (setq should-scan t))
+         ((progn (set-buffer tags-file-buf)
+                 (etags-regen--remove-tag file-name))
+          (setq should-scan t))))
+      (when (and should-scan
+                 (not (cl-some
+                       (lambda (ignore)
+                         (string-match-p
+                          (etags-regen--ignore-regexp ignore)
+                          relname))
+                       ignores)))
+        (etags-regen--append-tags file-name)
+        (let ((save-silently t)
+              (message-log-max nil))
+          (save-buffer 0))))))
+
+(defun etags-regen--remove-tag (file-name)
+  (goto-char (point-min))
+  (when (search-forward (format "\f\n%s," file-name) nil t)
+    (let ((start (match-beginning 0)))
+      (search-forward "\f\n" nil 'move)
+      (let ((inhibit-read-only t))
+        (delete-region start
+                       (if (eobp)
+                           (point)
+                         (- (point) 2)))))
+    t))
+
+(defun etags-regen--append-tags (&rest file-names)
+  (goto-char (point-max))
+  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
+        (inhibit-read-only t))
+    ;; XXX: call-process is significantly faster, though.
+    ;; Like 10ms vs 20ms here.  But `shell-command' makes it easy to
+    ;; direct stderr to a separate buffer.
+    (shell-command
+     (format "%s %s %s -o -"
+             etags-regen-program (mapconcat #'identity options " ")
+             (mapconcat #'identity file-names " "))
+     t etags-regen--errors-buffer-name))
+  ;; FIXME: Is there a better way to do this?
+  ;; Completion table is the only remaining place where the
+  ;; update is not incremental.
+  (setq-default tags-completion-table nil))
+
+(defun etags-regen--mark-as-new ()
+  (when (and etags-regen--tags-root
+             (not buffer-file-number))
+    (setq-local etags-regen--new-file t)))
+
+(defun etags-regen--tags-cleanup ()
+  (when etags-regen--tags-file
+    (let ((buffer (get-file-buffer etags-regen--tags-file)))
+      (and buffer
+           (kill-buffer buffer)))
+    (tags-reset-tags-tables)
+    (setq tags-file-name nil
+          tags-table-list nil
+          etags-regen--tags-file nil
+          etags-regen--tags-root nil))
+  (remove-hook 'after-save-hook #'etags-regen--update-file)
+  (remove-hook 'before-save-hook #'etags-regen--mark-as-new))
+
+(defvar etags-regen-mode-map (make-sparse-keymap))
+
+;;;###autoload
+(define-minor-mode etags-regen-mode
+  "Minor mode to automatically generate and update tags tables.
+
+This minor mode generates the tags table automatically based on
+the current project configuration, and later updates it as you
+edit the files and save the changes."
+  :global t
+  (if etags-regen-mode
+      (progn
+        (advice-add 'etags--xref-backend :before
+                    #'etags-regen--maybe-generate)
+        (advice-add 'tags-completion-at-point-function :before
+                    #'etags-regen--maybe-generate))
+    (advice-remove 'etags--xref-backend #'etags-regen--maybe-generate)
+    (advice-remove 'tags-completion-at-point-function #'etags-regen--maybe-generate)
+    (etags-regen--tags-cleanup)))
+
+(provide 'etags-regen)
+
+;;; etags-regen.el ends here

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

>> Should this be marked unsafe?  Actually, same question for all of the
>> above defcustoms, given that we use `shell-command-to-string'.
>
> Given that this option doesn't have the 'safe-local-variable' property,
> do we need to do something else?

I was thinking of setting

    :risky t

>>> +;; We have to list all extensions: etags falls back to Fortran
>>> +;; when it cannot determine the type of the file.
>>
>> (A battle-tested default, if nothing else.  ;-)
>
> I'd rather etags skipped unknown files, but there is no such option.

Yeah, that might make more sense.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 02/01/2024 02:40, Stefan Kangas wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>>> Should this be marked unsafe?  Actually, same question for all of the
>>> above defcustoms, given that we use `shell-command-to-string'.
>>
>> Given that this option doesn't have the 'safe-local-variable' property,
>> do we need to do something else?
> 
> I was thinking of setting
> 
>      :risky t

I don't know, it seems like the users might want to set etags options 
from .dir-locals.el too. And the above would prompt the user every time 
they visit a file inside such project.

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Sun, 31 Dec 2023 01:43:25 +0200
> Cc: 67687@debbugs.gnu.org, eskinjp@gmail.com, stefankangas@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >>>> +;;;###autoload
> >>>> +(put 'etags-regen-file-extensions 'safe-local-variable
> >>>> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
> >>>
> >>> Why not use list-of-strings-p here?
> >>
> >> Again, that "core ELPA" consideration. We could deploy this feature to a
> >> number of released Emacs versions, if we don't introduce such dependencies.
> > 
> > Isn't this covered by the compat package on ELPA?  If not, I think it
> > should be.
> 
> These forms go into generated autoloads file for each installed package 
> (*-autoloads.el). I think compat doesn't make list-of-string-p 
> autoloaded, and autoloads files don't usually have (require ...) forms.
> 
> So while I haven't really tested this and could be missing something, it 
> seems brittle to rely on 'compat' for this function (if at all possible).

It is also extremely ugly to have those large functions in a bundled
package, when we already have list-of-strings-p in Emacs 29.  So how
about defining list-of-strings-p in etags-regen for older versions,
and then using it in the safe-local-variable property?  Or some other
solution to make this more elegant.  Wanting to let users use this
with older Emacsen has its limits, and IMNSHO this one crosses that
limit.

> Latest revision attached. Any further comments?

None from me, thanks.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 31/12/2023 09:07, Eli Zaretskii wrote:
>> Date: Sun, 31 Dec 2023 01:43:25 +0200
>> Cc: 67687@debbugs.gnu.org, eskinjp@gmail.com, stefankangas@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>>>> +;;;###autoload
>>>>>> +(put 'etags-regen-file-extensions 'safe-local-variable
>>>>>> +     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
>>>>>
>>>>> Why not use list-of-strings-p here?
>>>>
>>>> Again, that "core ELPA" consideration. We could deploy this feature to a
>>>> number of released Emacs versions, if we don't introduce such dependencies.
>>>
>>> Isn't this covered by the compat package on ELPA?  If not, I think it
>>> should be.
>>
>> These forms go into generated autoloads file for each installed package
>> (*-autoloads.el). I think compat doesn't make list-of-string-p
>> autoloaded, and autoloads files don't usually have (require ...) forms.
>>
>> So while I haven't really tested this and could be missing something, it
>> seems brittle to rely on 'compat' for this function (if at all possible).
> 
> It is also extremely ugly to have those large functions in a bundled
> package, when we already have list-of-strings-p in Emacs 29.  So how
> about defining list-of-strings-p in etags-regen for older versions,
> and then using it in the safe-local-variable property?

Even that wouldn't work, I think, for the same reason: the autoloads 
file doesn't load the related package eagerly. Though I suppose we could 
force one specific function definition into autoloads.

> Or some other
> solution to make this more elegant.  Wanting to let users use this
> with older Emacsen has its limits, and IMNSHO this one crosses that
> limit.

We already do this in project.el for one variable, too.

If you like, we could simplify the forms further, though, dropping the 
string-match-p checks. Forms for etags-regen-file-extensions and 
etags-regen-ignores would look like the one for 
project-vc-extra-root-markers.

The form for etags-regen-lang-regexp-alist is not a lists-of-string-p to 
begin with, so it will be more complex (it's "alist of lists of strings").

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Fixed a typo in dir-locals and implemented better support for
> etags-regen-ignores (though with one omission).
>
> To me it looks good to check in now.

Thanks for working on this long overdue feature.

Do we really need a minor mode for this, or should it just be a
defcustom in etags?  We have tons of minor modes for this or that
various minor feature, and the list in e.g. `C-h m' is really starting
to look really long, at least in my use.  I wonder whether that is
really justified.

> diff --git a/.dir-locals.el b/.dir-locals.el
> index e087aa89cd1..ce7febca851 100644
> --- a/.dir-locals.el
> +++ b/.dir-locals.el
> @@ -8,6 +8,12 @@
>           (vc-git-annotate-switches . "-w")
>           (bug-reference-url-format . "https://debbugs.gnu.org/%s")
>  	 (diff-add-log-use-relative-names . t)
> +         (etags-regen-regexp-alist
> +          .
> +          ((("c" "objc") .
> +            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
> +             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
> +         (etags-regen-ignores . ("test/manual/etags/"))

I'm not terribly familiar with how etags is implemented, so please
forgive me if some of these questions are naive.

Is there any way around having to do so much manual setup to get this to
work?  The above regexp is rather complex.

And is the idea that users will customize this by themselves?  Is it
feasible to shift most of that work to major mode authors [perhaps only
partially]?

> diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
> new file mode 100644
> index 00000000000..88b730195c3
> --- /dev/null
> +++ b/lisp/progmodes/etags-regen.el
> @@ -0,0 +1,411 @@
> +;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
> +
> +;; Copyright (C) 2021, 2023 Free Software Foundation, Inc.

Using just 2021-2023 here is fine.

> +;;; Commentary:
> +
> +;; Simple automatic tags generation with updates on save.
> +;;
> +;; The goal of this mode is to provide a feature that should be
> +;; familiar to the users of certain lightweight programmer's editors,
> +;; such as Sublime Text.  Which is "go to definition" with automatic
> +;; indexing, added in ST3 (released in 2017).

This makes it sound like we're just copying others, when we could be
more confident.  Emacs has had the described feature since before 2017.
I propose dropping all references to Sublime Text and reducing the above
to simply saying:

    This library provides automatic indexing for Emacs "go to
    definition" feature, the `xref-go-forward' command (bound to `M-.'
    by default).

> +;; At the moment reindexing works off before/after-save-hook, but to
> +;; handle more complex changes (e.g. the user switching to another

(We usually prefer "for example" to "e.g.".)

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 30/12/2023 00:29, Stefan Kangas wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> Fixed a typo in dir-locals and implemented better support for
>> etags-regen-ignores (though with one omission).
>>
>> To me it looks good to check in now.
> 
> Thanks for working on this long overdue feature.
> 
> Do we really need a minor mode for this, or should it just be a
> defcustom in etags?  We have tons of minor modes for this or that
> various minor feature, and the list in e.g. `C-h m' is really starting
> to look really long, at least in my use.  I wonder whether that is
> really justified.

As long as it's in a separate file, it should be easy to publish it to 
ELPA (as "core" package). Which is an option that's nice to have, even 
if not essential. Sometime later, when the features and implementation 
stabilize, we could merge the files, leaving the code in ELPA for older 
emacsen. Or something like that.

>> diff --git a/.dir-locals.el b/.dir-locals.el
>> index e087aa89cd1..ce7febca851 100644
>> --- a/.dir-locals.el
>> +++ b/.dir-locals.el
>> @@ -8,6 +8,12 @@
>>            (vc-git-annotate-switches . "-w")
>>            (bug-reference-url-format . "https://debbugs.gnu.org/%s")
>>   	 (diff-add-log-use-relative-names . t)
>> +         (etags-regen-regexp-alist
>> +          .
>> +          ((("c" "objc") .
>> +            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
>> +             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
>> +         (etags-regen-ignores . ("test/manual/etags/"))
> 
> I'm not terribly familiar with how etags is implemented, so please
> forgive me if some of these questions are naive.
> 
> Is there any way around having to do so much manual setup to get this to
> work?  The above regexp is rather complex.
> 
> And is the idea that users will customize this by themselves?  Is it
> feasible to shift most of that work to major mode authors [perhaps only
> partially]?

This is only necessary for language constructs not supported by etags 
OOtB. Such as our C macros which define Elisp functions and variables. 
These are the same regexps that we have in our Makefile.

So this is a per-project thing, rather than per-language. Most users and 
projects shouldn't need it, or wouldn't need it right away.

>> diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
>> new file mode 100644
>> index 00000000000..88b730195c3
>> --- /dev/null
>> +++ b/lisp/progmodes/etags-regen.el
>> @@ -0,0 +1,411 @@
>> +;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
>> +
>> +;; Copyright (C) 2021, 2023 Free Software Foundation, Inc.
> 
> Using just 2021-2023 here is fine.

Ok.

>> +;;; Commentary:
>> +
>> +;; Simple automatic tags generation with updates on save.
>> +;;
>> +;; The goal of this mode is to provide a feature that should be
>> +;; familiar to the users of certain lightweight programmer's editors,
>> +;; such as Sublime Text.  Which is "go to definition" with automatic
>> +;; indexing, added in ST3 (released in 2017).
> 
> This makes it sound like we're just copying others, when we could be
> more confident.  Emacs has had the described feature since before 2017.
> I propose dropping all references to Sublime Text and reducing the above
> to simply saying:

But... it didn't? Otherwise you wouldn't have called it "long overdue", 
right?

Anyway, I'm not married to the above text, it's just a description of 
how I'm thinking about the problem. But I would invite you and other to 
consider how the ST users take advantage of automatic indexing without 
having to be aware of how information is stored behind the scenes (tag 
files or not), when considering the sections of the manual touching on 
etags-regen-mode.

>      This library provides automatic indexing for Emacs "go to
>      definition" feature, the `xref-go-forward' command (bound to `M-.'
>      by default).

Sure.

We could also add some text that would distinguish it from the general 
notion of "automatic indexing", so that the users of Eglot, for example, 
don't consider it necessary to enable this mode. Even though they would 
also want indexing to remain automatic.

>> +;; At the moment reindexing works off before/after-save-hook, but to
>> +;; handle more complex changes (e.g. the user switching to another
> 
> (We usually prefer "for example" to "e.g.".)

No problem.

Though searching across the codebase, the number of hits for these two 
options seems to be about the same (5K vs 4K).

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

> As long as it's in a separate file, it should be easy to publish it to
> ELPA (as "core" package). Which is an option that's nice to have, even
> if not essential. Sometime later, when the features and implementation
> stabilize, we could merge the files, leaving the code in ELPA for older
> emacsen. Or something like that.

That overall plan sounds good to me, thanks.  We'll have to decide on
the details as we go of course, but clearly it makes sense to allow the
feature some breathing room to stabilize.

> This is only necessary for language constructs not supported by etags
> OOtB. Such as our C macros which define Elisp functions and variables.
> These are the same regexps that we have in our Makefile.
>
> So this is a per-project thing, rather than per-language. Most users and
> projects shouldn't need it, or wouldn't need it right away.

Ah, that makes more sense to me now.  Thank you.

Would it be helpful to put that explanation in the .dir-locals.el file
itself?

>>> +;;; Commentary:
>>> +
>>> +;; Simple automatic tags generation with updates on save.
>>> +;;
>>> +;; The goal of this mode is to provide a feature that should be
>>> +;; familiar to the users of certain lightweight programmer's editors,
>>> +;; such as Sublime Text.  Which is "go to definition" with automatic
>>> +;; indexing, added in ST3 (released in 2017).
>>
>> This makes it sound like we're just copying others, when we could be
>> more confident.  Emacs has had the described feature since before 2017.
>> I propose dropping all references to Sublime Text and reducing the above
>> to simply saying:
>
> But... it didn't? Otherwise you wouldn't have called it "long overdue",
> right?

Uhm, yeah, that could have been more clear.  I must have hatched a key
sentence when editing, or something.  Please let me try again.

The proposed text seemed to open up for the misunderstanding that "go to
definition" is a new feature, that Sublime text introduced in 2017 and
Emacs will now get in version 30.1.

I think we should clarify that the new feature is only "automatic
indexing".  Furthermore, doing things for the user in the background is
hardly revolutionary enough that we need to give Sublime text the credit
for the invention, or anything like that.  It's rather mundane these
days, as far as features go.  Users have learned to expect it.

This is what makes the feature long overdue.

Does that make more sense?

> Anyway, I'm not married to the above text, it's just a description of
> how I'm thinking about the problem. But I would invite you and other to
> consider how the ST users take advantage of automatic indexing without
> having to be aware of how information is stored behind the scenes (tag
> files or not), when considering the sections of the manual touching on
> etags-regen-mode.
>
>>      This library provides automatic indexing for Emacs "go to
>>      definition" feature, the `xref-go-forward' command (bound to `M-.'
>>      by default).
>
> Sure.
>
> We could also add some text that would distinguish it from the general
> notion of "automatic indexing", so that the users of Eglot, for example,
> don't consider it necessary to enable this mode. Even though they would
> also want indexing to remain automatic.

Indeed, the possible confusion with eglot could bear some documenting.
Perhaps we should add a new paragraph to the commentary explaining how
this feature will (or will not) interact with Eglot.

>>> +;; At the moment reindexing works off before/after-save-hook, but to
>>> +;; handle more complex changes (e.g. the user switching to another
>>
>> (We usually prefer "for example" to "e.g.".)
>
> No problem.
>
> Though searching across the codebase, the number of hits for these two
> options seems to be about the same (5K vs 4K).

Search for "abbreviations" in (info "(elisp) Documentation Tips").

But when we made that addition, we didn't bother changing all existing
documentation.  IIRC, most people in that discussion preferred a more
gradual approach.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 30/12/2023 22:31, Stefan Kangas wrote:

>> This is only necessary for language constructs not supported by etags
>> OOtB. Such as our C macros which define Elisp functions and variables.
>> These are the same regexps that we have in our Makefile.
>>
>> So this is a per-project thing, rather than per-language. Most users and
>> projects shouldn't need it, or wouldn't need it right away.
> 
> Ah, that makes more sense to me now.  Thank you.
> 
> Would it be helpful to put that explanation in the .dir-locals.el file
> itself?

.dir-locals.el already usually hosts per-project settings. And most 
users of this feature probably aren't going to read Emacs's one.

I've added another sentence to the docstring, let's see if it helps.

>>>> +;;; Commentary:
>>>> +
>>>> +;; Simple automatic tags generation with updates on save.
>>>> +;;
>>>> +;; The goal of this mode is to provide a feature that should be
>>>> +;; familiar to the users of certain lightweight programmer's editors,
>>>> +;; such as Sublime Text.  Which is "go to definition" with automatic
>>>> +;; indexing, added in ST3 (released in 2017).
>>>
>>> This makes it sound like we're just copying others, when we could be
>>> more confident.  Emacs has had the described feature since before 2017.
>>> I propose dropping all references to Sublime Text and reducing the above
>>> to simply saying:
>>
>> But... it didn't? Otherwise you wouldn't have called it "long overdue",
>> right?
> 
> Uhm, yeah, that could have been more clear.  I must have hatched a key
> sentence when editing, or something.  Please let me try again.
> 
> The proposed text seemed to open up for the misunderstanding that "go to
> definition" is a new feature, that Sublime text introduced in 2017 and
> Emacs will now get in version 30.1.

You might be right. The file/feature only contains the automatic 
indexing part, though.

> I think we should clarify that the new feature is only "automatic
> indexing".  Furthermore, doing things for the user in the background is
> hardly revolutionary enough that we need to give Sublime text the credit
> for the invention, or anything like that.  It's rather mundane these
> days, as far as features go.  Users have learned to expect it.
> 
> This is what makes the feature long overdue.

Yeah, it's hardly an innovation, more like in the "why don't we have 
this yet" department. But while automatic indexing has been around for a 
while, having it OOtB in lightweight editors wasn't commonplace. So as I 
recall it for ST3 (first beta in 2013, release in 2017) it was a 
meaningful step forward. The complex IDEs already had this for a long 
time, of course (but each was more specialized, and worked with a 
smaller number of languages).

>> We could also add some text that would distinguish it from the general
>> notion of "automatic indexing", so that the users of Eglot, for example,
>> don't consider it necessary to enable this mode. Even though they would
>> also want indexing to remain automatic.
> 
> Indeed, the possible confusion with eglot could bear some documenting.
> Perhaps we should add a new paragraph to the commentary explaining how
> this feature will (or will not) interact with Eglot.

Suggestions welcome. I'm not sure how to phrase that without mentioning 
etags, tags files, and xref backends (in general and the names of 
specific ones).

>>>> +;; At the moment reindexing works off before/after-save-hook, but to
>>>> +;; handle more complex changes (e.g. the user switching to another
>>>
>>> (We usually prefer "for example" to "e.g.".)
>>
>> No problem.
>>
>> Though searching across the codebase, the number of hits for these two
>> options seems to be about the same (5K vs 4K).
> 
> Search for "abbreviations" in (info "(elisp) Documentation Tips").
> 
> But when we made that addition, we didn't bother changing all existing
> documentation.  IIRC, most people in that discussion preferred a more
> gradual approach.

All right, I've replaced the two "e.g."'s in user-facing text.

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 30/12/2023 22:31, Stefan Kangas wrote:
>
>> Would it be helpful to put that explanation in the .dir-locals.el file
>> itself?
>
> .dir-locals.el already usually hosts per-project settings. And most
> users of this feature probably aren't going to read Emacs's one.

I was mostly thinking about us poor Emacs maintainers, but either way is
fine by me.

> Yeah, it's hardly an innovation, more like in the "why don't we have
> this yet" department. But while automatic indexing has been around for a
> while, having it OOtB in lightweight editors wasn't commonplace. So as I
> recall it for ST3 (first beta in 2013, release in 2017) it was a
> meaningful step forward. The complex IDEs already had this for a long
> time, of course (but each was more specialized, and worked with a
> smaller number of languages).

OK, that's interesting, as far as text editor history goes.

Still, I'm hesitant to give them too much acknowledgement for what
basically amounts to no longer being among the worst in class.  As you
say, IDEs have already been doing this type of thing for a long time.
Sublime Text is non-free software too, which doesn't do much to make me
happier about mentioning their name.

But if you think it's a useful piece of history, then by all means let's
keep it.  Perhaps it could be moved to a separate history section rather
than the introductory paragraph, though?  It's your call.

>> Indeed, the possible confusion with eglot could bear some documenting.
>> Perhaps we should add a new paragraph to the commentary explaining how
>> this feature will (or will not) interact with Eglot.
>
> Suggestions welcome. I'm not sure how to phrase that without mentioning
> etags, tags files, and xref backends (in general and the names of
> specific ones).

The most pressing thing to explain, I think, is what happens if you run
both this mode and eglot.  Users will want to run the global mode but
still use eglot for some projects.

I don't really have a concrete suggestion, as I don't have a clear idea
of how it works.  :-)  But I think eglot will just take over and the
etags stuff will be ignored, no?

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 31/12/2023 01:25, Stefan Kangas wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> On 30/12/2023 22:31, Stefan Kangas wrote:
>>
>>> Would it be helpful to put that explanation in the .dir-locals.el file
>>> itself?
>>
>> .dir-locals.el already usually hosts per-project settings. And most
>> users of this feature probably aren't going to read Emacs's one.
> 
> I was mostly thinking about us poor Emacs maintainers, but either way is
> fine by me.

Speaking of maintainers, I'm curious if I'll ever see the day when 'make 
tags' outputs "Use 'M-x etags-regen-mode' instead" ;-)

>> Yeah, it's hardly an innovation, more like in the "why don't we have
>> this yet" department. But while automatic indexing has been around for a
>> while, having it OOtB in lightweight editors wasn't commonplace. So as I
>> recall it for ST3 (first beta in 2013, release in 2017) it was a
>> meaningful step forward. The complex IDEs already had this for a long
>> time, of course (but each was more specialized, and worked with a
>> smaller number of languages).
> 
> OK, that's interesting, as far as text editor history goes.
> 
> Still, I'm hesitant to give them too much acknowledgement for what
> basically amounts to no longer being among the worst in class.  As you
> say, IDEs have already been doing this type of thing for a long time.
> Sublime Text is non-free software too, which doesn't do much to make me
> happier about mentioning their name.

GNU software has a long history of taking inspiration from non-free 
software, though.

> But if you think it's a useful piece of history, then by all means let's
> keep it.  Perhaps it could be moved to a separate history section rather
> than the introductory paragraph, though?  It's your call.

Nah, let's keep your alternative. I might mention it somewhere later, 
e.g. in a blog post.

>>> Indeed, the possible confusion with eglot could bear some documenting.
>>> Perhaps we should add a new paragraph to the commentary explaining how
>>> this feature will (or will not) interact with Eglot.
>>
>> Suggestions welcome. I'm not sure how to phrase that without mentioning
>> etags, tags files, and xref backends (in general and the names of
>> specific ones).
> 
> The most pressing thing to explain, I think, is what happens if you run
> both this mode and eglot.  Users will want to run the global mode but
> still use eglot for some projects.

Or lsp-mode, or cider, or a bunch of other Xref and completion backends 
that are still around but are less popular than LSP.

Oh BTW elisp-mode will also continue to use its own backends, unaffected 
by etags-regen-mode (unless xref-etags-mode is on), even if we decide to 
mention only the built-in solutions.

> I don't really have a concrete suggestion, as I don't have a clear idea
> of how it works.  :-)  But I think eglot will just take over and the
> etags stuff will be ignored, no?

In Eglot-managed buffers -- yes.

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Sun, 31 Dec 2023 01:58:55 +0200
> Cc: 67687@debbugs.gnu.org, eskinjp@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 31/12/2023 01:25, Stefan Kangas wrote:
> > Dmitry Gutov <dmitry@gutov.dev> writes:
> > 
> >> On 30/12/2023 22:31, Stefan Kangas wrote:
> >>
> >>> Would it be helpful to put that explanation in the .dir-locals.el file
> >>> itself?
> >>
> >> .dir-locals.el already usually hosts per-project settings. And most
> >> users of this feature probably aren't going to read Emacs's one.
> > 
> > I was mostly thinking about us poor Emacs maintainers, but either way is
> > fine by me.
> 
> Speaking of maintainers, I'm curious if I'll ever see the day when 'make 
> tags' outputs "Use 'M-x etags-regen-mode' instead" ;-)

I don't yet see why we'd need that.  As one data point, my TAGS files
in the Emacs repository were generated in Feb 2023, and I still use
them almost every day without any visible problems.  And for Lisp
code, M-. doesn't use TAGS by default anyway.

Unlike "indexing" in other IDEs, the Emacs tags commands are well
equipped to cope with changes in sources, and don't fail
catastrophically when there are such changes, as long as functions and
variables don't move between files.  The auto-regen mode might be
needed for some projects where source files change significantly at
high pace, but not in Emacs, at least not IME.

So whether this mode should be turned on by default is something that
is yet to be seen.  Let's not hurry and make decisions in haste,
certainly not wrt the use of this as part of Emacs maintenance.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 31/12/2023 09:23, Eli Zaretskii wrote:
>> Date: Sun, 31 Dec 2023 01:58:55 +0200
>> Cc: 67687@debbugs.gnu.org, eskinjp@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 31/12/2023 01:25, Stefan Kangas wrote:
>>> Dmitry Gutov <dmitry@gutov.dev> writes:
>>>
>>>> On 30/12/2023 22:31, Stefan Kangas wrote:
>>>>
>>>>> Would it be helpful to put that explanation in the .dir-locals.el file
>>>>> itself?
>>>>
>>>> .dir-locals.el already usually hosts per-project settings. And most
>>>> users of this feature probably aren't going to read Emacs's one.
>>>
>>> I was mostly thinking about us poor Emacs maintainers, but either way is
>>> fine by me.
>>
>> Speaking of maintainers, I'm curious if I'll ever see the day when 'make
>> tags' outputs "Use 'M-x etags-regen-mode' instead" ;-)
> 
> I don't yet see why we'd need that.

There is no hard requirement indeed.

> As one data point, my TAGS files
> in the Emacs repository were generated in Feb 2023, and I still use
> them almost every day without any visible problems.  And for Lisp
> code, M-. doesn't use TAGS by default anyway.
> 
> Unlike "indexing" in other IDEs, the Emacs tags commands are well
> equipped to cope with changes in sources, and don't fail
> catastrophically when there are such changes, as long as functions and
> variables don't move between files.  The auto-regen mode might be
> needed for some projects where source files change significantly at
> high pace, but not in Emacs, at least not IME.

The upside is not having to generate tags to begin with (e.g. for new 
contributors, or new repository checkouts/worktrees/etc), and not 
worrying about keeping them updated even when you add or remove a 
function. No matter how rare, that still happens.

There are some downsides too (e.g. some runtime overhead), let's wait 
and see if those turn out to be noticeable enough.

> So whether this mode should be turned on by default is something that
> is yet to be seen.  Let's not hurry and make decisions in haste,
> certainly not wrt the use of this as part of Emacs maintenance.

I'm not proposing any haste in that respect.

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

>>    . there are no updates for NEWS and the Emacs manual
>
> I'll certainly add something to NEWS. Not sure where and what should be
> in the manual.

How about a new node describing this feature under `(emacs) Tags
Tables`?

> Which brings a problem: the mode is now likely unusable over Tramp in
> any project of significant size. Something to improve later.

Should this be documented somewhere?

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 30/12/2023 00:17, Stefan Kangas wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>>>     . there are no updates for NEWS and the Emacs manual
>>
>> I'll certainly add something to NEWS. Not sure where and what should be
>> in the manual.
> 
> How about a new node describing this feature under `(emacs) Tags
> Tables`?

Good suggestion, but that still leaves the problem of organizing the text.

Do you want, perchance, to give it a try yourself?

Optimally, it might need a significant rewrite: we wouldn't point the 
user to 'etags' right away, or the "Create Tags Table" section.

The average user taking advantage of etags-regen-mode might not even 
need to know what a "tags table" is (or only know that in very broad 
strokes), so depending on our eventual approach we might drop that node, 
or rearrange the nodes in a particular order, prefacing that one 
("Create Tags Table") as the approach for advanced users.

>> Which brings a problem: the mode is now likely unusable over Tramp in
>> any project of significant size. Something to improve later.
> 
> Should this be documented somewhere?

Maybe, maybe not. The previous solution using xargs and stat (as still 
implemented on the branch -- now outdated) should work for most remote 
hosts (unixy ones). So it's likely a matter of having a user interested 
in this and giving them a short enough patch to try.

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 30/12/2023 00:17, Stefan Kangas wrote:
>> Dmitry Gutov <dmitry@gutov.dev> writes:
>>
>>>>     . there are no updates for NEWS and the Emacs manual
>>>
>>> I'll certainly add something to NEWS. Not sure where and what should be
>>> in the manual.
>>
>> How about a new node describing this feature under `(emacs) Tags
>> Tables`?
>
> Good suggestion, but that still leaves the problem of organizing the text.
>
> Do you want, perchance, to give it a try yourself?
>
> Optimally, it might need a significant rewrite: we wouldn't point the
> user to 'etags' right away, or the "Create Tags Table" section.
>
> The average user taking advantage of etags-regen-mode might not even
> need to know what a "tags table" is (or only know that in very broad
> strokes), so depending on our eventual approach we might drop that node,
> or rearrange the nodes in a particular order, prefacing that one
> ("Create Tags Table") as the approach for advanced users.

Indeed, all good points.  In an ideal world, "tags tables" would be an
implementation detail, perhaps only of interest to users with highly
specific and customized workflows.

I won't have time to work on a proper patch right now, but perhaps
something along these lines (warning: a *very* rough draft):

diff --git a/doc/emacs/maintaining.texi b/doc/emacs/maintaining.texi
index 0725d889747..f3d09b48f12 100644
--- a/doc/emacs/maintaining.texi
+++ b/doc/emacs/maintaining.texi
@@ -2666,6 +2666,9 @@ Tags Tables
 by using one of the commands from other packages that can produce such
 tables in the same format.)

+  Instead of creating a tags table manually, you might want to have it
+created automatically for you.  @xref{Update Tags Table}.
+
   Emacs uses the tags tables via the @code{etags} package as one of
 the supported backends for @code{xref}.  Because tags tables are
 produced by the @command{etags} command that is part of an Emacs
@@ -2683,7 +2686,8 @@ Tags Tables

 @menu
 * Tag Syntax::          Tag syntax for various types of code and text files.
-* Create Tags Table::   Creating a tags table with @command{etags}.
+* Update Tags Table::   Creating a tags table automatically.
+* Create Tags Table::   Creating a tags table manually with @command{etags}.
 * Etags Regexps::       Create arbitrary tags using regular expressions.
 @end menu

@@ -2877,8 +2881,21 @@ Tag Syntax
   You can also generate tags based on regexp matching (@pxref{Etags
 Regexps}) to handle other formats and languages.

+@node Update Tags Table
+@subsubsection Keeping Tags Tables up to date
+
+   The easiest way to manage tags tables is by enabling
+@code{etags-regen-mode} (@kbd{M-x etags-regen-mode RET}).  This minor
+mode generates the tags table automatically based on the current
+project configuration, and later updates it as you edit the files and
+save the changes.
+
+Note that this feature disables itself if you have already manually
+visited a tags table (with @kbd{M-x visit-tags-table}, or through an
+explicit prompt triggered by some feature that requires tags).
+
 @node Create Tags Table
-@subsubsection Creating Tags Tables
+@subsubsection Creating Tags Tables Manually
 @cindex @command{etags} program

   The @command{etags} program is used to create a tags table file.  It knows

>>> Which brings a problem: the mode is now likely unusable over Tramp in
>>> any project of significant size. Something to improve later.
>>
>> Should this be documented somewhere?
>
> Maybe, maybe not. The previous solution using xargs and stat (as still
> implemented on the branch -- now outdated) should work for most remote
> hosts (unixy ones). So it's likely a matter of having a user interested
> in this and giving them a short enough patch to try.

Right.  I basically never use Tramp, so I'm leaving this to others.
Copying in Michael in case he has any comments.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 30/12/2023 22:56, Stefan Kangas wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> On 30/12/2023 00:17, Stefan Kangas wrote:
>>> Dmitry Gutov<dmitry@gutov.dev>  writes:
>>>
>>>>>      . there are no updates for NEWS and the Emacs manual
>>>> I'll certainly add something to NEWS. Not sure where and what should be
>>>> in the manual.
>>> How about a new node describing this feature under `(emacs) Tags
>>> Tables`?
>> Good suggestion, but that still leaves the problem of organizing the text.
>>
>> Do you want, perchance, to give it a try yourself?
>>
>> Optimally, it might need a significant rewrite: we wouldn't point the
>> user to 'etags' right away, or the "Create Tags Table" section.
>>
>> The average user taking advantage of etags-regen-mode might not even
>> need to know what a "tags table" is (or only know that in very broad
>> strokes), so depending on our eventual approach we might drop that node,
>> or rearrange the nodes in a particular order, prefacing that one
>> ("Create Tags Table") as the approach for advanced users.
> Indeed, all good points.  In an ideal world, "tags tables" would be an
> implementation detail, perhaps only of interest to users with highly
> specific and customized workflows.
> 
> I won't have time to work on a proper patch right now, but perhaps
> something along these lines (warning: a*very*  rough draft):
> 
> diff --git a/doc/emacs/maintaining.texi b/doc/emacs/maintaining.texi
> index 0725d889747..f3d09b48f12 100644
> --- a/doc/emacs/maintaining.texi
> +++ b/doc/emacs/maintaining.texi
> @@ -2666,6 +2666,9 @@ Tags Tables
>   by using one of the commands from other packages that can produce such
>   tables in the same format.)
> 
> +  Instead of creating a tags table manually, you might want to have it
> +created automatically for you.  @xref{Update Tags Table}.
> +
>     Emacs uses the tags tables via the @code{etags} package as one of
>   the supported backends for @code{xref}.  Because tags tables are
>   produced by the @command{etags} command that is part of an Emacs
> @@ -2683,7 +2686,8 @@ Tags Tables
> 
>   @menu
>   * Tag Syntax::          Tag syntax for various types of code and text files.
> -* Create Tags Table::   Creating a tags table with @command{etags}.
> +* Update Tags Table::   Creating a tags table automatically.
> +* Create Tags Table::   Creating a tags table manually with @command{etags}.
>   * Etags Regexps::       Create arbitrary tags using regular expressions.
>   @end menu
> 
> @@ -2877,8 +2881,21 @@ Tag Syntax
>     You can also generate tags based on regexp matching (@pxref{Etags
>   Regexps}) to handle other formats and languages.
> 
> +@node Update Tags Table
> +@subsubsection Keeping Tags Tables up to date
> +
> +   The easiest way to manage tags tables is by enabling
> +@code{etags-regen-mode} (@kbd{M-x etags-regen-mode RET}).  This minor
> +mode generates the tags table automatically based on the current
> +project configuration, and later updates it as you edit the files and
> +save the changes.
> +
> +Note that this feature disables itself if you have already manually
> +visited a tags table (with @kbd{M-x visit-tags-table}, or through an
> +explicit prompt triggered by some feature that requires tags).
> +
>   @node Create Tags Table
> -@subsubsection Creating Tags Tables
> +@subsubsection Creating Tags Tables Manually
>   @cindex @command{etags} program
> 
>     The @command{etags} program is used to create a tags table file.  It knows

Thanks! This is already an improvement.

Regarding node names: if we call the new node "Update Tags Table", the 
users might want to visit it after "Create Tags Table" (to maybe see how 
to keep a tags table updated after creating it manually), and then read 
that this is not supported, as you note in the second paragraph.

We might implement something like that later, but it would have to be 
off by default, I think.

What if we call them "Automatic Tags Table" and "Manual Tags Table"? Or 
"Creating Tags Table Automatically" and "Creating Tags Table Manually"?

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

> What if we call them "Automatic Tags Table" and "Manual Tags Table"?

Much better, yes.

> Or "Creating Tags Table Automatically" and "Creating Tags Table
> Manually"?

Also better, but perhaps a bit long?  I think we try to keep node names
relatively short (IIUC, because the layout in `M-x info' doesn't play
well with long names).

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sat, 30 Dec 2023 12:56:38 -0800
> Cc: eskinjp@gmail.com, 67687@debbugs.gnu.org, 
> 	Michael Albinus <michael.albinus@gmx.de>
> 
>  @menu
>  * Tag Syntax::          Tag syntax for various types of code and text files.
> -* Create Tags Table::   Creating a tags table with @command{etags}.
> +* Update Tags Table::   Creating a tags table automatically.
> +* Create Tags Table::   Creating a tags table manually with @command{etags}.
>  * Etags Regexps::       Create arbitrary tags using regular expressions.
>  @end menu

Please don't change the order of the sections.  "Create" should
precede "update", since the regen feature is optional.  If this order
requires some changes in how the text is written, please make those
changes, but putting "update" before "create" is illogical.  Also, the
description of "update" says "create", which is another brow-raising
factor.

> +@node Update Tags Table
> +@subsubsection Keeping Tags Tables up to date
> +
> +   The easiest way to manage tags tables is by enabling
> +@code{etags-regen-mode} (@kbd{M-x etags-regen-mode RET}).

It is too early to say this, because if we do, people will ask why
that mode is not turned on by default.  We may turn it on by default
some day (and then the text will need to be rearranged or rewritten),
but we are not there yet, so let's please not put the cart before the
horse, and let's describe etags-regen-mode as what it is: an optional
minor mode that was just added, and should yet withstand the test of
time before we consider recommending it as _the_ primary MO.

> +Note that this feature disables itself if you have already manually
> +visited a tags table (with @kbd{M-x visit-tags-table}, or through an
> +explicit prompt triggered by some feature that requires tags).

This aspect is IMO somewhat problematic.  I wasn't aware of it, and
now that I read this, I'm not sure it is correct and will meet user
expectations.

bug#67687: Feature request: automatic tags management

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>>  @menu
>>  * Tag Syntax::          Tag syntax for various types of code and text files.
>> -* Create Tags Table::   Creating a tags table with @command{etags}.
>> +* Update Tags Table::   Creating a tags table automatically.
>> +* Create Tags Table::   Creating a tags table manually with @command{etags}.
>>  * Etags Regexps::       Create arbitrary tags using regular expressions.
>>  @end menu
>
> Please don't change the order of the sections.  "Create" should
> precede "update", since the regen feature is optional.  If this order
> requires some changes in how the text is written, please make those
> changes, but putting "update" before "create" is illogical.  Also, the
> description of "update" says "create", which is another brow-raising
> factor.

Nothing was moved, I think?  This sections is new.  :-)

I have no real opinion on the best order; Dmitriy's renamed node name is
"automatic tags table", which could come before or after "manual".

However, as you point out below, the feature is new and perhaps
shouldn't be mentioned first for that reason.

> It is too early to say this, because if we do, people will ask why
> that mode is not turned on by default.  We may turn it on by default
> some day (and then the text will need to be rearranged or rewritten),
> but we are not there yet, so let's please not put the cart before the
> horse, and let's describe etags-regen-mode as what it is: an optional
> minor mode that was just added, and should yet withstand the test of
> time before we consider recommending it as _the_ primary MO.

Makes sense, yes.  Note that my proposed text was written very quickly,
and intended more like a starting point for the structure than anything
else.  (It also doesn't document any user options, etc.)

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 31/12/2023 09:22, Stefan Kangas wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>>>   @menu
>>>   * Tag Syntax::          Tag syntax for various types of code and text files.
>>> -* Create Tags Table::   Creating a tags table with @command{etags}.
>>> +* Update Tags Table::   Creating a tags table automatically.
>>> +* Create Tags Table::   Creating a tags table manually with @command{etags}.
>>>   * Etags Regexps::       Create arbitrary tags using regular expressions.
>>>   @end menu
>> Please don't change the order of the sections.  "Create" should
>> precede "update", since the regen feature is optional.  If this order
>> requires some changes in how the text is written, please make those
>> changes, but putting "update" before "create" is illogical.  Also, the
>> description of "update" says "create", which is another brow-raising
>> factor.
> Nothing was moved, I think?  This sections is new.  🙂
> 
> I have no real opinion on the best order; Dmitriy's renamed node name is
> "automatic tags table", which could come before or after "manual".
> 
> However, as you point out below, the feature is new and perhaps
> shouldn't be mentioned first for that reason.

I'm happy to leave all the manual-related decisions to you guys.

If it shouldn't be in the manual initially -- no objections from me.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 31/12/2023 08:34, Eli Zaretskii wrote:
>> +Note that this feature disables itself if you have already manually
>> +visited a tags table (with @kbd{M-x visit-tags-table}, or through an
>> +explicit prompt triggered by some feature that requires tags).
> This aspect is IMO somewhat problematic.  I wasn't aware of it, and
> now that I read this, I'm not sure it is correct and will meet user
> expectations.

I'm pretty sure you asked for it: that even when this mode is on, it 
shouldn't interfere with completion tables explicitly visited by the user.

And either way it seems like a prerequisite for enabling 
etags-regen-mode by default sometimes in the future. Unless we decide to 
deprecate 'M-x visit-tags-table' at the same time.

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Sun, 31 Dec 2023 17:25:35 +0200
> Cc: eskinjp@gmail.com, 67687@debbugs.gnu.org, michael.albinus@gmx.de
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 31/12/2023 08:34, Eli Zaretskii wrote:
> >> +Note that this feature disables itself if you have already manually
> >> +visited a tags table (with @kbd{M-x visit-tags-table}, or through an
> >> +explicit prompt triggered by some feature that requires tags).
> > This aspect is IMO somewhat problematic.  I wasn't aware of it, and
> > now that I read this, I'm not sure it is correct and will meet user
> > expectations.
> 
> I'm pretty sure you asked for it: that even when this mode is on, it 
> shouldn't interfere with completion tables explicitly visited by the user.

"Interfere" and "prevent automatic regeneration" is not the same.

I think this probably warrants a separate defcustom: some people might
want such regeneration, even if the tags table was loaded manually,
others won't.  And I think the default should be to regenerate them
regardless.

> And either way it seems like a prerequisite for enabling 
> etags-regen-mode by default sometimes in the future.

How so?  The fact that I loaded TAGS doesn't necessarily mean I don't
want it updated when the sources change.  Or what am I missing?

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 31/12/2023 18:42, Eli Zaretskii wrote:
>> Date: Sun, 31 Dec 2023 17:25:35 +0200
>> Cc: eskinjp@gmail.com, 67687@debbugs.gnu.org, michael.albinus@gmx.de
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 31/12/2023 08:34, Eli Zaretskii wrote:
>>>> +Note that this feature disables itself if you have already manually
>>>> +visited a tags table (with @kbd{M-x visit-tags-table}, or through an
>>>> +explicit prompt triggered by some feature that requires tags).
>>> This aspect is IMO somewhat problematic.  I wasn't aware of it, and
>>> now that I read this, I'm not sure it is correct and will meet user
>>> expectations.
>>
>> I'm pretty sure you asked for it: that even when this mode is on, it
>> shouldn't interfere with completion tables explicitly visited by the user.
> 
> "Interfere" and "prevent automatic regeneration" is not the same.
> 
> I think this probably warrants a separate defcustom: some people might
> want such regeneration, even if the tags table was loaded manually,
> others won't.  And I think the default should be to regenerate them
> regardless.

Like mentioned previously, I think we'll get such an option sooner or 
later, but not in the first check-in. It merits an additional 
discussion, at least.

>> And either way it seems like a prerequisite for enabling
>> etags-regen-mode by default sometimes in the future.
> 
> How so?  The fact that I loaded TAGS doesn't necessarily mean I don't
> want it updated when the sources change.  Or what am I missing?

a) We won't add new files to the index, because we (apparently) can't 
simply use the project's list of files -- there is no guarantee that it 
matches the fileset that the original author of the TAGS file had in mind.

b) There is no way to pick up the --regex options used for generating 
the original TAGS, or any other options we don't know about. So if we 
were to just use the logic of regenerating tags for newly changed files, 
we would end up with a mix of tags in some files based on the set of 
--regex used in the past, and with tags for new files based on the 
configured set of --regex options.

Either way, we get a poorly-defined behavior with edge cases that are 
likely to surprise the user at different points of time. So we might 
indeed grow such a capability, but it'll probably stay off by default.

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Sun, 31 Dec 2023 19:53:27 +0200
> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>  michael.albinus@gmx.de
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> And either way it seems like a prerequisite for enabling
> >> etags-regen-mode by default sometimes in the future.
> > 
> > How so?  The fact that I loaded TAGS doesn't necessarily mean I don't
> > want it updated when the sources change.  Or what am I missing?
> 
> a) We won't add new files to the index, because we (apparently) can't 
> simply use the project's list of files -- there is no guarantee that it 
> matches the fileset that the original author of the TAGS file had in mind.

The user has etags-regen-ignores to control that.

> b) There is no way to pick up the --regex options used for generating 
> the original TAGS, or any other options we don't know about.

There are defcustoms to control both of those.

> Either way, we get a poorly-defined behavior with edge cases that are 
> likely to surprise the user at different points of time. So we might 
> indeed grow such a capability, but it'll probably stay off by default.

I agree that sometimes it could be against the user's expectations.
But I also think that other times it is according to user's
expectations.  Which tells me that this is a separate issue that needs
a separate knob; we shouldn't deterministically deduce what users want
in this respect from the fact that he/she loaded an existing tags
table.  Moreover, it is quite possible that even when the mode is
turned on, users might want sometimes to load tags tables manually.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 31/12/2023 21:27, Eli Zaretskii wrote:
>> Date: Sun, 31 Dec 2023 19:53:27 +0200
>> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>>   michael.albinus@gmx.de
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> And either way it seems like a prerequisite for enabling
>>>> etags-regen-mode by default sometimes in the future.
>>>
>>> How so?  The fact that I loaded TAGS doesn't necessarily mean I don't
>>> want it updated when the sources change.  Or what am I missing?
>>
>> a) We won't add new files to the index, because we (apparently) can't
>> simply use the project's list of files -- there is no guarantee that it
>> matches the fileset that the original author of the TAGS file had in mind.
> 
> The user has etags-regen-ignores to control that.

etags-regen-ignores wouldn't normally include filters that already are 
in project's ignores. But of course the user might choose to add them.

>> b) There is no way to pick up the --regex options used for generating
>> the original TAGS, or any other options we don't know about.
> 
> There are defcustoms to control both of those.

If all of these options are set to the expected values, what's stopping 
the user from calling 'M-x tags-reset-tags-tables' and having 
etags-regen-mode create the table on its own? The extra 10 seconds of 
waiting?

Also note that if you have an existing TAGS file in the root of the 
project, with default configuration etags-regen-mode will pick it up to 
visit and refresh it where necessary (when there are changed or missing 
files).

>> Either way, we get a poorly-defined behavior with edge cases that are
>> likely to surprise the user at different points of time. So we might
>> indeed grow such a capability, but it'll probably stay off by default.
> 
> I agree that sometimes it could be against the user's expectations.
> But I also think that other times it is according to user's
> expectations.  Which tells me that this is a separate issue that needs
> a separate knob; we shouldn't deterministically deduce what users want
> in this respect from the fact that he/she loaded an existing tags
> table.  Moreover, it is quite possible that even when the mode is
> turned on, users might want sometimes to load tags tables manually.

I don't mind adding additional knobs.

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Mon, 1 Jan 2024 03:23:36 +0200
> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>  michael.albinus@gmx.de
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> a) We won't add new files to the index, because we (apparently) can't
> >> simply use the project's list of files -- there is no guarantee that it
> >> matches the fileset that the original author of the TAGS file had in mind.
> > 
> > The user has etags-regen-ignores to control that.
> 
> etags-regen-ignores wouldn't normally include filters that already are 
> in project's ignores. But of course the user might choose to add them.
> 
> >> b) There is no way to pick up the --regex options used for generating
> >> the original TAGS, or any other options we don't know about.
> > 
> > There are defcustoms to control both of those.
> 
> If all of these options are set to the expected values, what's stopping 
> the user from calling 'M-x tags-reset-tags-tables' and having 
> etags-regen-mode create the table on its own? The extra 10 seconds of 
> waiting?
> 
> Also note that if you have an existing TAGS file in the root of the 
> project, with default configuration etags-regen-mode will pick it up to 
> visit and refresh it where necessary (when there are changed or missing 
> files).
> 
> >> Either way, we get a poorly-defined behavior with edge cases that are
> >> likely to surprise the user at different points of time. So we might
> >> indeed grow such a capability, but it'll probably stay off by default.
> > 
> > I agree that sometimes it could be against the user's expectations.
> > But I also think that other times it is according to user's
> > expectations.  Which tells me that this is a separate issue that needs
> > a separate knob; we shouldn't deterministically deduce what users want
> > in this respect from the fact that he/she loaded an existing tags
> > table.  Moreover, it is quite possible that even when the mode is
> > turned on, users might want sometimes to load tags tables manually.
> 
> I don't mind adding additional knobs.

I guess for now it will be enough to document that invoking
visit-tags-table disables the mode for that project.  AFAICT this is
not currently documented anywhere in the patch.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

On 01/01/2024 14:07, Eli Zaretskii wrote:
>>>> Either way, we get a poorly-defined behavior with edge cases that are
>>>> likely to surprise the user at different points of time. So we might
>>>> indeed grow such a capability, but it'll probably stay off by default.
>>> I agree that sometimes it could be against the user's expectations.
>>> But I also think that other times it is according to user's
>>> expectations.  Which tells me that this is a separate issue that needs
>>> a separate knob; we shouldn't deterministically deduce what users want
>>> in this respect from the fact that he/she loaded an existing tags
>>> table.  Moreover, it is quite possible that even when the mode is
>>> turned on, users might want sometimes to load tags tables manually.
>> I don't mind adding additional knobs.
> I guess for now it will be enough to document that invoking
> visit-tags-table disables the mode for that project.  AFAICT this is
> not currently documented anywhere in the patch.

OK, added a paragraph to the mode's docstring.

[-- Attachment #2: etags-regen-v7.diff --]
[-- Type: text/x-patch, Size: 18603 bytes --]

diff --git a/.dir-locals.el b/.dir-locals.el
index e087aa89cd1..ce7febca851 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -8,6 +8,12 @@
          (vc-git-annotate-switches . "-w")
          (bug-reference-url-format . "https://debbugs.gnu.org/%s")
 	 (diff-add-log-use-relative-names . t)
+         (etags-regen-regexp-alist
+          .
+          ((("c" "objc") .
+            ("/[ \t]*DEFVAR_[A-Z_ \t(]+\"\\([^\"]+\\)\"/\\1/"
+             "/[ \t]*DEFVAR_[A-Z_ \t(]+\"[^\"]+\",[ \t]\\([A-Za-z0-9_]+\\)/\\1/"))))
+         (etags-regen-ignores . ("test/manual/etags/"))
          (vc-prepare-patches-separately . nil)))
  (c-mode . ((c-file-style . "GNU")
             (c-noise-macro-names . ("INLINE" "NO_INLINE" "ATTRIBUTE_NO_SANITIZE_UNDEFINED"
diff --git a/etc/NEWS b/etc/NEWS
index ba40efa2e9d..0928ae5a2dc 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1270,6 +1270,11 @@ the needs of users with red-green or blue-yellow color deficiency.
 The Info manual "(modus-themes) Top" describes the details and
 showcases all their customization options.
 
+** New global minor mode 'etags-regen-mode'.
+This minor mode generates the tags table automatically based on the
+current project configuration, and later updates it as you edit the
+files and save the changes.
+
 \f
 * Incompatible Lisp Changes in Emacs 30.1
 
diff --git a/lisp/progmodes/etags-regen.el b/lisp/progmodes/etags-regen.el
new file mode 100644
index 00000000000..01bd626c962
--- /dev/null
+++ b/lisp/progmodes/etags-regen.el
@@ -0,0 +1,431 @@
+;;; etags-regen.el --- Auto-(re)regenerating tags  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2021-2023 Free Software Foundation, Inc.
+
+;; Author: Dmitry Gutov <dmitry@gutov.dev>
+;; Keywords: tools
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Simple automatic tags generation with updates on save.
+;;
+;; This mode provides automatic indexing for Emacs "go to definition"
+;; feature, the `xref-go-forward' command (bound to `M-.' by default).
+;;
+;; At the moment reindexing works off before/after-save-hook, but to
+;; handle more complex changes (for example, the user switching to
+;; another branch from the terminal) we can look into plugging into
+;; something like `filenotify'.
+;;
+;; Note that this feature disables itself if the user has some tags
+;; table already visited (with `M-x visit-tags-table', or through an
+;; explicit prompt triggered by some feature that requires tags).
+
+;;; Code:
+
+(require 'cl-lib)
+
+(defgroup etags-regen nil
+  "Auto-(re)generating tags."
+  :group 'tools)
+
+(defvar etags-regen--tags-file nil)
+(defvar etags-regen--tags-root nil)
+(defvar etags-regen--new-file nil)
+
+(declare-function project-root "project")
+(declare-function project-files "project")
+(declare-function dired-glob-regexp "dired")
+
+(defcustom etags-regen-program (executable-find "etags")
+  "Name of the etags program used by `etags-regen-mode'.
+
+If you only have `ctags' installed, you can also set this to
+\"ctags -e\".  Some features might not be supported this way."
+  ;; Always having our 'etags' here would be easier, but we can't
+  ;; always rely on it being installed.  So it might be ctags's etags.
+  :type 'file
+  :version "30.1")
+
+(defcustom etags-regen-tags-file "TAGS"
+  "Name of the tags file to create inside the project by `etags-regen-mode'.
+
+The value should either be a simple file name (no directory
+specified), or a function that accepts the project root directory
+and returns a distinct absolute file name for its tags file.  The
+latter possibility is useful when you prefer to store the tag
+files somewhere else, for example in `temporary-file-directory'."
+  :type '(choice (string :tag "File name")
+                 (function :tag "Function that returns file name"))
+  :version "30.1")
+
+(defcustom etags-regen-program-options nil
+  "List of additional options for etags program invoked by `etags-regen-mode'."
+  :type '(repeat string)
+  :version "30.1")
+
+(defcustom etags-regen-regexp-alist nil
+  "Mapping of languages to etags regexps for `etags-regen-mode'.
+
+These regexps are used in addition to the tags made with the
+standard parsing based on the language.
+
+The value must be a list where each element has the
+form (LANGUAGES . TAG-REGEXPS) where both LANGUAGES and
+TAG-REGEXPS are lists of strings.
+
+Each language should be one of the recognized by etags, see
+`etags --help'.  Each tag regexp should be a string in the format
+documented for the `--regex' arguments (without `{language}').
+
+We currently support only Emacs's etags program with this option."
+  :type '(repeat
+          (cons
+           :tag "Languages group"
+           (repeat (string :tag "Language name"))
+           (repeat (string :tag "Tag Regexp"))))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-regexp-alist 'safe-local-variable
+     (lambda (value)
+       (and (listp value)
+            (seq-every-p
+             (lambda (group)
+               (and (consp group)
+                    (listp (car group))
+                    (listp (cdr group))
+                    (seq-every-p #'stringp (car group))
+                    (seq-every-p #'stringp (cdr group))))
+             value))))
+
+;; We have to list all extensions: etags falls back to Fortran
+;; when it cannot determine the type of the file.
+;; http://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00323.html
+(defcustom etags-regen-file-extensions
+  '("rb" "js" "py" "pl" "el" "c" "cpp" "cc" "h" "hh" "hpp"
+    "java" "go" "cl" "lisp" "prolog" "php" "erl" "hrl"
+    "F" "f" "f90" "for" "cs" "a" "asm" "ads" "adb" "ada")
+  "Code file extensions for `etags-regen-mode'.
+
+File extensions to generate the tags for."
+  :type '(repeat (string :tag "File extension"))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-file-extensions 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+;; FIXME: We don't support root anchoring yet.
+(defcustom etags-regen-ignores nil
+  "Additional ignore rules, in the format of `project-ignores'."
+  :type '(repeat
+          (string :tag "Glob to ignore"))
+  :version "30.1")
+
+;;;###autoload
+(put 'etags-regen-ignores 'safe-local-variable
+     (lambda (value) (and (listp value) (seq-every-p #'stringp value))))
+
+(defvar etags-regen--errors-buffer-name "*etags-regen-tags-errors*")
+
+(defvar etags-regen--rescan-files-limit 100)
+
+(defun etags-regen--all-mtimes (proj)
+  (let ((files (etags-regen--all-files proj))
+        (mtimes (make-hash-table :test 'equal))
+        file-name-handler-alist)
+    (dolist (f files)
+      (condition-case nil
+          (puthash f
+                   (file-attribute-modification-time
+                    (file-attributes f))
+                   mtimes)
+        (file-missing nil)))
+    mtimes))
+
+(defun etags-regen--choose-tags-file (proj)
+  (if (functionp etags-regen-tags-file)
+      (funcall etags-regen-tags-file (project-root proj))
+    (expand-file-name etags-regen-tags-file (project-root proj))))
+
+(defun etags-regen--refresh (proj)
+  (save-excursion
+    (let* ((tags-file (etags-regen--choose-tags-file proj))
+           (tags-mtime (file-attribute-modification-time
+                        (file-attributes tags-file)))
+           (all-mtimes (etags-regen--all-mtimes proj))
+           added-files
+           changed-files
+           removed-files)
+      (etags-regen--visit-table tags-file (project-root proj))
+      (set-buffer (get-file-buffer tags-file))
+      (dolist (file (tags-table-files))
+        (let ((mtime (gethash file all-mtimes)))
+          (cond
+           ((null mtime)
+            (push file removed-files))
+           ((time-less-p tags-mtime mtime)
+            (push file changed-files)
+            (remhash file all-mtimes))
+           (t
+            (remhash file all-mtimes)))))
+      (maphash
+       (lambda (key _value)
+         (push key added-files))
+       all-mtimes)
+      (if (> (+ (length added-files)
+                (length changed-files)
+                (length removed-files))
+             etags-regen--rescan-files-limit)
+          (progn
+            (message "etags-regen: Too many changes, falling back to full rescan")
+            (etags-regen--tags-cleanup))
+        (dolist (file (nconc removed-files changed-files))
+          (etags-regen--remove-tag file))
+        (when (or changed-files added-files)
+          (apply #'etags-regen--append-tags
+                 (nconc changed-files added-files)))
+        (when (or changed-files added-files removed-files)
+          (let ((save-silently t)
+                (message-log-max nil))
+            (save-buffer 0)))))))
+
+(defun etags-regen--maybe-generate ()
+  (let (proj)
+    (when (and etags-regen--tags-root
+               (not (file-in-directory-p default-directory
+                                         etags-regen--tags-root)))
+      (etags-regen--tags-cleanup))
+    (when (and (not etags-regen--tags-root)
+               ;; If existing table is visited that's not generated by
+               ;; this mode, skip all functionality.
+               (not (or tags-file-name
+                        tags-table-list))
+               (file-exists-p (etags-regen--choose-tags-file
+                               (setq proj (project-current)))))
+      (message "Found existing tags table, refreshing...")
+      (etags-regen--refresh proj))
+    (when (and (not (or tags-file-name
+                        tags-table-list))
+               (setq proj (or proj (project-current))))
+      (message "Generating new tags table...")
+      (let ((start (time-to-seconds)))
+        (etags-regen--tags-generate proj)
+        (message "...done (%.2f s)" (- (time-to-seconds) start))))))
+
+(defun etags-regen--all-files (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         ;; TODO: Make the scanning more efficient, e.g. move the
+         ;; filtering by glob to project (project-files-filtered...).
+         (files (project-files proj))
+         (match-re (concat
+                    "\\."
+                    (regexp-opt etags-regen-file-extensions)
+                    "\\'"))
+         (ir-start (1- (length root)))
+         (ignores-regexps
+          (mapcar #'etags-regen--ignore-regexp
+                  etags-regen-ignores)))
+    (cl-delete-if
+     (lambda (f) (or (not (string-match-p match-re f))
+                (string-match-p "/\\.#" f) ;Backup files.
+                (cl-some (lambda (ignore) (string-match ignore f ir-start))
+                         ignores-regexps)))
+     files)))
+
+(defun etags-regen--ignore-regexp (ignore)
+  (require 'dired)
+  ;; It's somewhat brittle to rely on Dired.
+  (let ((re (dired-glob-regexp ignore)))
+    ;; We could implement root anchoring here, but \\= doesn't work in
+    ;; string-match :-(.
+    (concat (unless (eq ?/ (aref re 3)) "/")
+            ;; Cutting off the anchors added by `dired-glob-regexp'.
+            (substring re 2 (- (length re) 2))
+            ;; This way we allow a glob to match against a directory
+            ;; name, or a file name.  And when it ends with / already,
+            ;; no need to add the anchoring.
+            (unless (eq ?/ (aref re (- (length re) 3)))
+              ;; Either match a full name segment, or eos.
+              "\\(?:/\\|\\'\\)"))))
+
+(defun etags-regen--tags-generate (proj)
+  (let* ((root (project-root proj))
+         (default-directory root)
+         (files (etags-regen--all-files proj))
+         (tags-file (etags-regen--choose-tags-file proj))
+         (ctags-p (etags-regen--ctags-p))
+         (command (format "%s %s %s - -o %s"
+                          etags-regen-program
+                          (mapconcat #'identity
+                                     (etags-regen--build-program-options ctags-p)
+                                     " ")
+                          ;; ctags's etags requires '-L' for stdin input.
+                          (if ctags-p "-L" "")
+                          tags-file)))
+    (with-temp-buffer
+      (mapc (lambda (f)
+              (insert f "\n"))
+            files)
+      (shell-command-on-region (point-min) (point-max) command
+                               nil nil etags-regen--errors-buffer-name t))
+    (etags-regen--visit-table tags-file root)))
+
+(defun etags-regen--visit-table (tags-file root)
+  ;; Invalidate the scanned tags after any change is written to disk.
+  (add-hook 'after-save-hook #'etags-regen--update-file)
+  (add-hook 'before-save-hook #'etags-regen--mark-as-new)
+  (setq etags-regen--tags-file tags-file
+        etags-regen--tags-root root)
+  (visit-tags-table etags-regen--tags-file))
+
+(defun etags-regen--ctags-p ()
+  (string-search "Ctags"
+                 (shell-command-to-string
+                  (format "%s --version" etags-regen-program))))
+
+(defun etags-regen--build-program-options (ctags-p)
+  (when (and etags-regen-regexp-alist ctags-p)
+    (user-error "etags-regen-regexp-alist is not supported with Ctags"))
+  (nconc
+   (mapcan
+    (lambda (group)
+      (mapcan
+       (lambda (lang)
+         (mapcar (lambda (regexp)
+                   (concat "--regex="
+                           (shell-quote-argument
+                            (format "{%s}%s" lang regexp))))
+                 (cdr group)))
+       (car group)))
+    etags-regen-regexp-alist)
+   (mapcar #'shell-quote-argument
+           etags-regen-program-options)))
+
+(defun etags-regen--update-file ()
+  ;; TODO: Maybe only do this when Emacs is idle for a bit.  Or defer
+  ;; the updates and do them later in bursts when the table is used.
+  (let* ((file-name buffer-file-name)
+         (tags-file-buf (and etags-regen--tags-root
+                             (get-file-buffer etags-regen--tags-file)))
+         (relname (concat "/" (file-relative-name file-name
+                                                  etags-regen--tags-root)))
+         (ignores etags-regen-ignores)
+         pr should-scan)
+    (save-excursion
+      (when tags-file-buf
+        (cond
+         ((and etags-regen--new-file
+               (kill-local-variable 'etags-regen--new-file)
+               (setq pr (project-current))
+               (equal (project-root pr) etags-regen--tags-root)
+               (member file-name (project-files pr)))
+          (set-buffer tags-file-buf)
+          (setq should-scan t))
+         ((progn (set-buffer tags-file-buf)
+                 (etags-regen--remove-tag file-name))
+          (setq should-scan t))))
+      (when (and should-scan
+                 (not (cl-some
+                       (lambda (ignore)
+                         (string-match-p
+                          (etags-regen--ignore-regexp ignore)
+                          relname))
+                       ignores)))
+        (etags-regen--append-tags file-name)
+        (let ((save-silently t)
+              (message-log-max nil))
+          (save-buffer 0))))))
+
+(defun etags-regen--remove-tag (file-name)
+  (goto-char (point-min))
+  (when (search-forward (format "\f\n%s," file-name) nil t)
+    (let ((start (match-beginning 0)))
+      (search-forward "\f\n" nil 'move)
+      (let ((inhibit-read-only t))
+        (delete-region start
+                       (if (eobp)
+                           (point)
+                         (- (point) 2)))))
+    t))
+
+(defun etags-regen--append-tags (&rest file-names)
+  (goto-char (point-max))
+  (let ((options (etags-regen--build-program-options (etags-regen--ctags-p)))
+        (inhibit-read-only t))
+    ;; XXX: call-process is significantly faster, though.
+    ;; Like 10ms vs 20ms here.  But `shell-command' makes it easy to
+    ;; direct stderr to a separate buffer.
+    (shell-command
+     (format "%s %s %s -o -"
+             etags-regen-program (mapconcat #'identity options " ")
+             (mapconcat #'identity file-names " "))
+     t etags-regen--errors-buffer-name))
+  ;; FIXME: Is there a better way to do this?
+  ;; Completion table is the only remaining place where the
+  ;; update is not incremental.
+  (setq-default tags-completion-table nil))
+
+(defun etags-regen--mark-as-new ()
+  (when (and etags-regen--tags-root
+             (not buffer-file-number))
+    (setq-local etags-regen--new-file t)))
+
+(defun etags-regen--tags-cleanup ()
+  (when etags-regen--tags-file
+    (let ((buffer (get-file-buffer etags-regen--tags-file)))
+      (and buffer
+           (kill-buffer buffer)))
+    (tags-reset-tags-tables)
+    (setq tags-file-name nil
+          tags-table-list nil
+          etags-regen--tags-file nil
+          etags-regen--tags-root nil))
+  (remove-hook 'after-save-hook #'etags-regen--update-file)
+  (remove-hook 'before-save-hook #'etags-regen--mark-as-new))
+
+(defvar etags-regen-mode-map (make-sparse-keymap))
+
+;;;###autoload
+(define-minor-mode etags-regen-mode
+  "Minor mode to automatically generate and update tags tables.
+
+This minor mode generates the tags table automatically based on
+the current project configuration, and later updates it as you
+edit the files and save the changes.
+
+The exception is made when the user has already selected a tags
+table manually (for example, using \\[visit-tags-table]).  Then
+this mode doesn't do anything.  Reset the visited tags tables
+using \\[tags-reset-tags-tables]."
+  :global t
+  (if etags-regen-mode
+      (progn
+        (advice-add 'etags--xref-backend :before
+                    #'etags-regen--maybe-generate)
+        (advice-add 'tags-completion-at-point-function :before
+                    #'etags-regen--maybe-generate))
+    (advice-remove 'etags--xref-backend #'etags-regen--maybe-generate)
+    (advice-remove 'tags-completion-at-point-function #'etags-regen--maybe-generate)
+    (etags-regen--tags-cleanup)))
+
+(provide 'etags-regen)
+
+;;; etags-regen.el ends here

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Mon, 1 Jan 2024 17:47:09 +0200
> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>  michael.albinus@gmx.de
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > I guess for now it will be enough to document that invoking
> > visit-tags-table disables the mode for that project.  AFAICT this is
> > not currently documented anywhere in the patch.
> 
> OK, added a paragraph to the mode's docstring.
> [...]
> +
> +The exception is made when the user has already selected a tags
> +table manually (for example, using \\[visit-tags-table]).  Then
> +this mode doesn't do anything.  Reset the visited tags tables
> +using \\[tags-reset-tags-tables]."

The above is only true for the project whose TAGS was manually
visited, right?  IOW, it's not that once the use invokes
visit-tags-table once, this mode will be effectively disabled for the
entire session, right?

If so, I suggest to make it clear:

  If you select a tags table manually (for example, using
  \\[visit-tags-table]), then this mode will be effectively disabled
  for the project or the directory tree from which the tags table was
  visited (the mode will still auto-regenerate tags tables for other
  projects and directory trees).  Use \\[tags-reset-tags-tables] to
  countermand the effect of a previous \\[visit-tags-table].

Otherwise, this LGTM, thanks.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 01/01/2024 18:50, Eli Zaretskii wrote:
>> Date: Mon, 1 Jan 2024 17:47:09 +0200
>> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>>   michael.albinus@gmx.de
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> I guess for now it will be enough to document that invoking
>>> visit-tags-table disables the mode for that project.  AFAICT this is
>>> not currently documented anywhere in the patch.
>>
>> OK, added a paragraph to the mode's docstring.
>> [...]
>> +
>> +The exception is made when the user has already selected a tags
>> +table manually (for example, using \\[visit-tags-table]).  Then
>> +this mode doesn't do anything.  Reset the visited tags tables
>> +using \\[tags-reset-tags-tables]."
> 
> The above is only true for the project whose TAGS was manually
> visited, right?  IOW, it's not that once the use invokes
> visit-tags-table once, this mode will be effectively disabled for the
> entire session, right?

No, it's the latter. It's off for the whole session, just like a visited 
TAGS file is active for the whole session (not limited to any specific 
project).

The current choice is optimized for making it possible to flip 
etags-regen-mode on globally while not interrupting the workflows of 
most (hopefully all) people who currently use etags.

> If so, I suggest to make it clear:
> 
>    If you select a tags table manually (for example, using
>    \\[visit-tags-table]), then this mode will be effectively disabled
>    for the project or the directory tree from which the tags table was
>    visited (the mode will still auto-regenerate tags tables for other
>    projects and directory trees).  Use \\[tags-reset-tags-tables] to
>    countermand the effect of a previous \\[visit-tags-table].
> 
> Otherwise, this LGTM, thanks.

We could do something like this too, but a tags file, in general, can't 
always be attributed to a specific project. At least we won't always be 
able to do that correctly for custom-made tag files. For example, the 
user could have that file include another tags file from a different 
directory on the same level, just like we here do with directories 'src' 
and 'lisp'. If 'src' and 'lisp' were configured to be separate projects, 
a visited src/TAGS would be attributed to the 'src' project only, while 
in fact it covers both. So the simplest solution is not to try.

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Mon, 1 Jan 2024 19:23:28 +0200
> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>  michael.albinus@gmx.de
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 01/01/2024 18:50, Eli Zaretskii wrote:
> >> +The exception is made when the user has already selected a tags
> >> +table manually (for example, using \\[visit-tags-table]).  Then
> >> +this mode doesn't do anything.  Reset the visited tags tables
> >> +using \\[tags-reset-tags-tables]."
> > 
> > The above is only true for the project whose TAGS was manually
> > visited, right?  IOW, it's not that once the use invokes
> > visit-tags-table once, this mode will be effectively disabled for the
> > entire session, right?
> 
> No, it's the latter. It's off for the whole session, just like a visited 
> TAGS file is active for the whole session (not limited to any specific 
> project).

OK, then please tweak my suggested text to explain the situation
correctly instead.

> a tags file, in general, can't always be attributed to a specific
> project.

That's too bad, IMO, but it's a separate issue.  One of the annoyances
I had with tags was that it was difficult to have several independent
tags tables loaded, each one for a different set of sources.  I hoped
that project.el has solved this already, but I guess we are not there
yet.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 01/01/2024 19:39, Eli Zaretskii wrote:
>> Date: Mon, 1 Jan 2024 19:23:28 +0200
>> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>>   michael.albinus@gmx.de
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 01/01/2024 18:50, Eli Zaretskii wrote:
>>>> +The exception is made when the user has already selected a tags
>>>> +table manually (for example, using \\[visit-tags-table]).  Then
>>>> +this mode doesn't do anything.  Reset the visited tags tables
>>>> +using \\[tags-reset-tags-tables]."
>>>
>>> The above is only true for the project whose TAGS was manually
>>> visited, right?  IOW, it's not that once the use invokes
>>> visit-tags-table once, this mode will be effectively disabled for the
>>> entire session, right?
>>
>> No, it's the latter. It's off for the whole session, just like a visited
>> TAGS file is active for the whole session (not limited to any specific
>> project).
> 
> OK, then please tweak my suggested text to explain the situation
> correctly instead.

How about this?

If you select a tags table manually (for example, using
\\[visit-tags-table]), then this mode will be effectively
disabled for the entire session.  Use \\[tags-reset-tags-tables]
to countermand the effect of a previous \\[visit-tags-table].

>> a tags file, in general, can't always be attributed to a specific
>> project.
> 
> That's too bad, IMO, but it's a separate issue.  One of the annoyances
> I had with tags was that it was difficult to have several independent
> tags tables loaded, each one for a different set of sources.  I hoped
> that project.el has solved this already, but I guess we are not there
> yet.

etags-regen-mode kind of solves that, by checking and resynchronizing 
tags tables when you switch between projects. That works when a tags 
file already exists with the recognized name in that other project (e.g. 
because you worked on it just recently).

Projectile has a different solution using local tags tables, which is 
easy to adapt to project.el:

(defun project-visit-tags ()
   (let* ((pr (project-current))
          (tags-file (and pr
                          (expand-file-name "TAGS" (project-root pr)))))
     (and tags-file
          (file-exists-p tags-file)
          (visit-tags-table tags-file t))))

(add-hook 'find-file-hook #'project-visit-tags)

And if you limit the above to only certain projects (by e.g. checking 
for some additional file in the directory), then etags-regen-mode should 
stay active in the other projects, because it won't see the buffer-local 
bindings for tags-file-name and tags-table-list (the above snippet keeps 
the global value nil).

The buffer-local tag tables are not that well-tested overall (e.g. even 
tags-reset-tags-tables resets the local binding, if available, rather 
than the global one), but this combination should work in a personal config.

bug#67687: Feature request: automatic tags management

[Eli Zaretskii]

> Date: Mon, 1 Jan 2024 20:48:37 +0200
> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>  michael.albinus@gmx.de
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 01/01/2024 19:39, Eli Zaretskii wrote:
> >> Date: Mon, 1 Jan 2024 19:23:28 +0200
> >> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
> >>   michael.albinus@gmx.de
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >>
> >> On 01/01/2024 18:50, Eli Zaretskii wrote:
> >>>> +The exception is made when the user has already selected a tags
> >>>> +table manually (for example, using \\[visit-tags-table]).  Then
> >>>> +this mode doesn't do anything.  Reset the visited tags tables
> >>>> +using \\[tags-reset-tags-tables]."
> >>>
> >>> The above is only true for the project whose TAGS was manually
> >>> visited, right?  IOW, it's not that once the use invokes
> >>> visit-tags-table once, this mode will be effectively disabled for the
> >>> entire session, right?
> >>
> >> No, it's the latter. It's off for the whole session, just like a visited
> >> TAGS file is active for the whole session (not limited to any specific
> >> project).
> > 
> > OK, then please tweak my suggested text to explain the situation
> > correctly instead.
> 
> How about this?
> 
> If you select a tags table manually (for example, using
> \\[visit-tags-table]), then this mode will be effectively
> disabled for the entire session.  Use \\[tags-reset-tags-tables]
> to countermand the effect of a previous \\[visit-tags-table].

SGTM, thanks.

> (defun project-visit-tags ()
>    (let* ((pr (project-current))
>           (tags-file (and pr
>                           (expand-file-name "TAGS" (project-root pr)))))
>      (and tags-file
>           (file-exists-p tags-file)
>           (visit-tags-table tags-file t))))
> 
> (add-hook 'find-file-hook #'project-visit-tags)

This could work, but IMO it would be more elegant to avoid use of
find-file-hook, and instead have some data structure that maps
projects or directories to tags table files.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 01/01/2024 21:25, Eli Zaretskii wrote:
>> Date: Mon, 1 Jan 2024 20:48:37 +0200
>> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>>   michael.albinus@gmx.de
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 01/01/2024 19:39, Eli Zaretskii wrote:
>>>> Date: Mon, 1 Jan 2024 19:23:28 +0200
>>>> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>>>>    michael.albinus@gmx.de
>>>> From: Dmitry Gutov <dmitry@gutov.dev>
>>>>
>>>> On 01/01/2024 18:50, Eli Zaretskii wrote:
>>>>>> +The exception is made when the user has already selected a tags
>>>>>> +table manually (for example, using \\[visit-tags-table]).  Then
>>>>>> +this mode doesn't do anything.  Reset the visited tags tables
>>>>>> +using \\[tags-reset-tags-tables]."
>>>>>
>>>>> The above is only true for the project whose TAGS was manually
>>>>> visited, right?  IOW, it's not that once the use invokes
>>>>> visit-tags-table once, this mode will be effectively disabled for the
>>>>> entire session, right?
>>>>
>>>> No, it's the latter. It's off for the whole session, just like a visited
>>>> TAGS file is active for the whole session (not limited to any specific
>>>> project).
>>>
>>> OK, then please tweak my suggested text to explain the situation
>>> correctly instead.
>>
>> How about this?
>>
>> If you select a tags table manually (for example, using
>> \\[visit-tags-table]), then this mode will be effectively
>> disabled for the entire session.  Use \\[tags-reset-tags-tables]
>> to countermand the effect of a previous \\[visit-tags-table].
> 
> SGTM, thanks.

All right. I think I'll wait a days or two for any more comments and 
then install the feature.

>> (defun project-visit-tags ()
>>     (let* ((pr (project-current))
>>            (tags-file (and pr
>>                            (expand-file-name "TAGS" (project-root pr)))))
>>       (and tags-file
>>            (file-exists-p tags-file)
>>            (visit-tags-table tags-file t))))
>>
>> (add-hook 'find-file-hook #'project-visit-tags)
> 
> This could work, but IMO it would be more elegant to avoid use of
> find-file-hook, and instead have some data structure that maps
> projects or directories to tags table files.

Looking into it, the variable default-tags-table-function seemed like it 
could work for this purpose, with settings like this:

(defun my/project-default-tags-table ()
   (let* ((pr (project-current))
          (tags-file (and pr
                          (expand-file-name "TAGS" (project-root pr)))))
     (and tags-file
          (file-exists-p tags-file)
          tags-file)))

(setq
  tags-table-list '("~/vc/emacs-master/") ; List with all your projects
  default-tags-table-function #'my/project-default-tags-table)

But alas it only changes which table is used first for finding a 
definition (before all others are tried), and all completions are 
collected from all completion tables in a global variable 
(tags-completion-table). So there is more work to be done to support 
simultaneously loaded independent tags tables. The new user option could 
look like this one, but it would also be used for the case of CONT=t, 
and tags-completion-table would need to be set buffer-locally in tags 
table buffers.

Note that etags-regen-mode switches between the tables automatically 
already, so this would be orthogonal functionality at best. Though 
hopefully compatible, and something that'd help avoid rescanning the 
completions table when switching between projects.

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 02/01/2024 03:40, Dmitry Gutov wrote:
> On 01/01/2024 21:25, Eli Zaretskii wrote:
>>> Date: Mon, 1 Jan 2024 20:48:37 +0200
>>> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>>> michael.albinus@gmx.de
>>> From: Dmitry Gutov <dmitry@gutov.dev>
>>>
>>> On 01/01/2024 19:39, Eli Zaretskii wrote:
>>>>> Date: Mon, 1 Jan 2024 19:23:28 +0200
>>>>> Cc: stefankangas@gmail.com, eskinjp@gmail.com, 67687@debbugs.gnu.org,
>>>>> michael.albinus@gmx.de
>>>>> From: Dmitry Gutov <dmitry@gutov.dev>
>>>>>
>>>>> On 01/01/2024 18:50, Eli Zaretskii wrote:
>>>>>>> +The exception is made when the user has already selected a tags
>>>>>>> +table manually (for example, using \\[visit-tags-table]).  Then
>>>>>>> +this mode doesn't do anything.  Reset the visited tags tables
>>>>>>> +using \\[tags-reset-tags-tables]."
>>>>>>
>>>>>> The above is only true for the project whose TAGS was manually
>>>>>> visited, right?  IOW, it's not that once the use invokes
>>>>>> visit-tags-table once, this mode will be effectively disabled for the
>>>>>> entire session, right?
>>>>>
>>>>> No, it's the latter. It's off for the whole session, just like a 
>>>>> visited
>>>>> TAGS file is active for the whole session (not limited to any specific
>>>>> project).
>>>>
>>>> OK, then please tweak my suggested text to explain the situation
>>>> correctly instead.
>>>
>>> How about this?
>>>
>>> If you select a tags table manually (for example, using
>>> \\[visit-tags-table]), then this mode will be effectively
>>> disabled for the entire session.  Use \\[tags-reset-tags-tables]
>>> to countermand the effect of a previous \\[visit-tags-table].
>>
>> SGTM, thanks.
> 
> All right. I think I'll wait a days or two for any more comments and 
> then install the feature.

Pushed to master, and closing. Thanks to everyone for the discussion.

bug#67687: Feature request: automatic tags management

[Francesco Potortì]

>a) We won't add new files to the index, because we (apparently) can't 
>simply use the project's list of files -- there is no guarantee that it 
>matches the fileset that the original author of the TAGS file had in mind.
>
>b) There is no way to pick up the --regex options used for generating 
>the original TAGS, or any other options we don't know about. So if we 
>were to just use the logic of regenerating tags for newly changed files, 
>we would end up with a mix of tags in some files based on the set of 
>--regex used in the past, and with tags for new files based on the 
>configured set of --regex options.

Maybe the TAGS files should contain information to regenerate them?

bug#67687: Feature request: automatic tags management

[Dmitry Gutov]

On 02/01/2024 12:41, Francesco Potortì wrote:
>> a) We won't add new files to the index, because we (apparently) can't
>> simply use the project's list of files -- there is no guarantee that it
>> matches the fileset that the original author of the TAGS file had in mind.
>>
>> b) There is no way to pick up the --regex options used for generating
>> the original TAGS, or any other options we don't know about. So if we
>> were to just use the logic of regenerating tags for newly changed files,
>> we would end up with a mix of tags in some files based on the set of
>> --regex used in the past, and with tags for new files based on the
>> configured set of --regex options.
> Maybe the TAGS files should contain information to regenerate them?

It would ideally need something more nuanced, like a list of globs to 
match and a list of globs to ignore, so that, for example, new files 
could also be added to the index.

And also the directory structure(s) involved (see my previous example of 
two dirs side-by-side)... overall, this seems like a difficult direction 
to proceed when we have an automatic approach for generating tags with 
its own settings already.