bug#45443: 28.0.50; Can't find definition of compilation--message->loc

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Drew Adams]

> It's a general problem with uses of cl-defstruct and similar
> constructs: they generate functions and macros that the Help functions
> are unable to find.

Yep.  It's a big doc/help problem, IMO.  And
we've moved more and more stuff to things like
`cl-defstruct' (which in itself is fine).

I don't see a good general solution to this
problem.  But it is, I think, something
important for the quintessential
"self-documenting" editor.  I truly hope that
some smart and ambitious hacker tackles this.

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Unknown]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

Drew Adams <drew.adams@oracle.com> writes:

>> It's a general problem with uses of cl-defstruct and similar
>> constructs: they generate functions and macros that the Help functions
>> are unable to find.
>
> Yep.  It's a big doc/help problem, IMO.  And
> we've moved more and more stuff to things like
> `cl-defstruct' (which in itself is fine).
>
> I don't see a good general solution to this
> problem.  But it is, I think, something
> important for the quintessential
> "self-documenting" editor.  I truly hope that
> some smart and ambitious hacker tackles this.

One possible approach is, if the regular expression code fails to find a
location, we can fall back to expand macros until we find the definition
(a defalias in the case of a function, or a defvar in the case of a
variable), or we reach the end of the file.

I attach a first implementation of this approach, with some tests for
the function definition that Richard wanted to find.  I'm sure there are
still missing cases that are interesting (closures?).  Give it a try and
see if there are still relevant symbols whose definition Emacs is unable
to locate.  I'm also open to feedback about the macro expansion logic.
Could it be more efficient?  Thanks.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Improve-find-definition-in-Help-buffers.patch --]
[-- Type: text/x-patch, Size: 6061 bytes --]

From 34662c47b359792e048bb7020d7eb0fbf94e7838 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Daniel=20Mart=C3=ADn?= <mardani29@yahoo.es>
Date: Sun, 27 Dec 2020 01:07:26 +0100
Subject: [PATCH] Improve "find definition" in *Help* buffers

*
lisp/emacs-lisp/find-func.el (find-function--search-by-expanding-macros):
New internal function that searches for a function or variable by
expanding macros in a buffer.
* lisp/emacs-lisp/find-func.el (find-function-search-for-symbol): If
our regexp algorithm could not find a location for the symbol
definition, resort to find-function--search-by-expanding-macros.
* test/lisp/emacs-lisp/find-func-tests.el: Add a automatic test for a
function and variable generated by a macro.
* etc/NEWS: Advertise the improved functionality.
---
 etc/NEWS                                |  6 +++
 lisp/emacs-lisp/find-func.el            | 65 ++++++++++++++++++++++++-
 test/lisp/emacs-lisp/find-func-tests.el | 10 ++++
 3 files changed, 80 insertions(+), 1 deletion(-)

diff --git a/etc/NEWS b/etc/NEWS
index a320acb5fa..2bfaccd2ef 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -114,6 +114,12 @@ choosing a group, or clicking a button in the "*Help*" buffers when
 looking at the doc string of a function that belongs to one of these
 groups.
 
+---
+** Improved "find definition" feature of *Help* buffers.
+Now clicking on the link to find the definition of functions generated
+by 'cl-defstruct', or variables generated by 'define-derived-mode',
+for example, will go to the exact place where they are defined.
+
 ** New variable 'redisplay-skip-initial-frame' to enable batch redisplay tests.
 Setting it to nil forces the redisplay to do its job even in the
 initial frame used in batch mode.
diff --git a/lisp/emacs-lisp/find-func.el b/lisp/emacs-lisp/find-func.el
index 074e7db295..7796a72ecf 100644
--- a/lisp/emacs-lisp/find-func.el
+++ b/lisp/emacs-lisp/find-func.el
@@ -389,7 +389,70 @@ find-function-search-for-symbol
                   (progn
                     (beginning-of-line)
                     (cons (current-buffer) (point)))
-                (cons (current-buffer) nil)))))))))
+                ;; If the regexp search didn't find the location of
+                ;; the symbol (for example, because it is generated by
+                ;; a macro), try a slightly more expensive search that
+                ;; expands macros until it finds the symbol.
+                (cons (current-buffer)
+                      (find-function--search-by-expanding-macros
+                       (current-buffer) symbol type))))))))))
+
+(defun find-function--try-macroexpand (form)
+  "Try to macroexpand FORM in full or partially.
+This is a best-effort operation in which if macroexpansion fails,
+this function returns FORM as is."
+  (ignore-errors
+    (or
+     (macroexpand-all form)
+     (macroexpand-1 form)
+     form)))
+
+(defun find-function--any-subform-p (form pred)
+  "Walk FORM and apply PRED to its subexpressions.
+Return t if any PRED returns t."
+  (cond
+   ((not (consp form)) nil)
+   ((funcall pred form) t)
+   (t
+    (cl-destructuring-bind (left-child . right-child) form
+      (or
+       (find-function--any-subform-p left-child pred)
+       (find-function--any-subform-p right-child pred))))))
+
+(defun find-function--search-by-expanding-macros (buf symbol type)
+  "Expand macros in BUF to search for the definition of SYMBOL of TYPE."
+  (catch 'found
+    (with-current-buffer buf
+      (save-excursion
+        (goto-char (point-min))
+        (condition-case nil
+            (while t
+              (let ((form (read (current-buffer)))
+                    (expected-symbol-p
+                      (lambda (form)
+                        (cond
+                         ((null type)
+                          ;; Check if a given form is a `defalias' to
+                          ;; SYM, the function name we are searching
+                          ;; for.  All functions in Emacs Lisp
+                          ;; ultimately expand to a `defalias' form
+                          ;; after several steps of macroexpansion.
+                          (and (eq (car-safe form) 'defalias)
+                               (equal (car-safe (cdr form))
+                                      `(quote ,symbol))))
+                         ((eq type 'defvar)
+                          ;; Variables generated by macros ultimately
+                          ;; expand to `defvar'.
+                          (and (eq (car-safe form) 'defvar)
+                               (eq (car-safe (cdr form)) symbol)))
+                         (t nil)))))
+                (when (find-function--any-subform-p
+                       (find-function--try-macroexpand form)
+                       expected-symbol-p)
+                  ;; We want to return the location at the beginning
+                  ;; of the macro, so move back one sexp.
+                  (throw 'found (progn (backward-sexp) (point))))))
+          (end-of-file nil))))))
 
 (defun find-function-library (function &optional lisp-only verbose)
   "Return the pair (ORIG-FUNCTION . LIBRARY) for FUNCTION.
diff --git a/test/lisp/emacs-lisp/find-func-tests.el b/test/lisp/emacs-lisp/find-func-tests.el
index d77eb6757f..03df4bb9ff 100644
--- a/test/lisp/emacs-lisp/find-func-tests.el
+++ b/test/lisp/emacs-lisp/find-func-tests.el
@@ -43,5 +43,15 @@ find-func-tests--library-completion
                      (concat data-directory (kbd "n x / TAB RET"))
                    (read-library-name)))))
 
+;; Avoid a byte-compilation warning that may confuse people reading
+;; the result of the following test.
+(declare-function compilation--message->loc nil "compile")
+
+(ert-deftest find-func-tests--locate-macro-generated-symbols () ;bug#45443
+  (should (cdr (find-function-search-for-symbol
+                #'compilation--message->loc nil "compile")))
+  (should (cdr (find-function-search-for-symbol
+                'c-mode-hook 'defvar "cc-mode"))))
+
 (provide 'find-func-tests)
 ;;; find-func-tests.el ends here
-- 
2.28.0

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Lars Ingebrigtsen]

Unknown <unknown@unknown.invalid> writes:

> I attach a first implementation of this approach, with some tests for
> the function definition that Richard wanted to find.  I'm sure there are
> still missing cases that are interesting (closures?).  Give it a try and
> see if there are still relevant symbols whose definition Emacs is unable
> to locate.  I'm also open to feedback about the macro expansion logic.
> Could it be more efficient?  Thanks.

This is something I've wanted for a long time, so I went ahead and
pushed it to Emacs 28.  :-)  I tried it on a couple cl-defstructs, and
it seems to work well for me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Eli Zaretskii]

> From: Daniel Martín <mardani29@yahoo.es>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  45443@debbugs.gnu.org
> Date: Sun, 27 Dec 2020 01:51:29 +0100
> 
> One possible approach is, if the regular expression code fails to find a
> location, we can fall back to expand macros until we find the definition
> (a defalias in the case of a function, or a defvar in the case of a
> variable), or we reach the end of the file.

Why do we need to expand macros? isn't it enough to find the defstruct
itself, by looking for a partial match?

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Daniel Martín <mardani29@yahoo.es>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  45443@debbugs.gnu.org
>> Date: Sun, 27 Dec 2020 01:51:29 +0100
>> 
>> One possible approach is, if the regular expression code fails to find a
>> location, we can fall back to expand macros until we find the definition
>> (a defalias in the case of a function, or a defvar in the case of a
>> variable), or we reach the end of the file.
>
> Why do we need to expand macros? isn't it enough to find the defstruct
> itself, by looking for a partial match?

I haven't look at the patch, but I think the approach of macro expanding
is more general as should be able to track any function definition that
is synthesized by any macro.

  Andrea

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Daniel Martín <mardani29@yahoo.es>,  rms@gnu.org,
>   45443@debbugs.gnu.org
> Date: Sun, 27 Dec 2020 17:59:09 +0000
> 
> > Why do we need to expand macros? isn't it enough to find the defstruct
> > itself, by looking for a partial match?
> 
> I haven't look at the patch, but I think the approach of macro expanding
> is more general as should be able to track any function definition that
> is synthesized by any macro.

It is also more expensive and complicated.

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Unknown]

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

>>
>> Why do we need to expand macros? isn't it enough to find the defstruct
>> itself, by looking for a partial match?
>
> I haven't look at the patch, but I think the approach of macro expanding
> is more general as should be able to track any function definition that
> is synthesized by any macro.
>

Yes, my patch tried a more general approach, which would not only find
function definitions, but also defvars like the hooks that are
synthesized by define-major-mode, for example.

There's some opportunities to do less work, though.  For example, I
think it does not make sense to expand defuns because those were handled
in a previous step.  I think that'd reduce the search space
significantly.

Another possible approach for this problem is to search textually for
just the things that we're typically interested in (like, cl-defstruct
or define-derived-mode), and expand only those to see if they synthesize
the symbol we are looking for.  It will be a less general solution, but
it will be faster.  We may add more cases in the future, if needed.

Thoughts?

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Unknown]

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

>>
>> Why do we need to expand macros? isn't it enough to find the defstruct
>> itself, by looking for a partial match?
>
> I haven't look at the patch, but I think the approach of macro expanding
> is more general as should be able to track any function definition that
> is synthesized by any macro.
>

Yes, my patch tried a more general approach, which would not only find
function definitions, but also defvars like the hooks that are
synthesized by define-major-mode, for example.

There's some opportunities to do less work, though.  For example, I
think it does not make sense to expand defuns because those were handled
in a previous step.  I think that'd reduce the search space
significantly.

Another possible approach for this problem is to search textually for
just the things that we're typically interested in (like, cl-defstruct
or define-derived-mode), and expand only those to see if they synthesize
the symbol we are looking for.  It will be a less general solution, but
it will be faster.  We may add more cases in the future, if needed.

Thoughts?

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Daniel Martín via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
> text editors" <bug-gnu-emacs@gnu.org> writes:
>
>>>
>>> Why do we need to expand macros? isn't it enough to find the defstruct
>>> itself, by looking for a partial match?
>>
>> I haven't look at the patch, but I think the approach of macro expanding
>> is more general as should be able to track any function definition that
>> is synthesized by any macro.
>>
>
> Yes, my patch tried a more general approach, which would not only find
> function definitions, but also defvars like the hooks that are
> synthesized by define-major-mode, for example.
>
> There's some opportunities to do less work, though.  For example, I
> think it does not make sense to expand defuns because those were handled
> in a previous step.  I think that'd reduce the search space
> significantly.
>
> Another possible approach for this problem is to search textually for
> just the things that we're typically interested in (like, cl-defstruct
> or define-derived-mode), and expand only those to see if they synthesize
> the symbol we are looking for.  It will be a less general solution, but
> it will be faster.  We may add more cases in the future, if needed.
>
> Thoughts?

I personally like the generic approach.  Also considered should be
running only when the regexp based one is failing and therefore with no
performance hit in most cases but only fixing the broken ones.

  Andrea

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Richard Stallman]

Trying to debug a bug in compile-goto-error on Rmail files,
I typed C-h f compilation--message->loc RET
then clicked on the file name -- which is supposed to find the definition
of that function.

It did not find the definition; instead it said,

    Unable to find location in file

I also tried M-x find-function RET compilation--message->loc RET.
It found a call to compilation--message->loc, not the definition.

I searched for that name in the file and did not find a definition.
I will try grepping for it.



In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 2.24.32, cairo version 1.15.10)
 of 2020-12-08 built on freetop
Repository revision: 0155bd0fdb166c97a2ce76cc5bc64fd195a676d3
Repository branch: master
System Description: Trisquel GNU/Linux Etiona (9.0)

Configured using:
 'configure --with-gnutls=ifavailable 'CFLAGS=-O0 -g''

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK2 X11 XDBE XIM MODULES THREADS PDUMPER

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Help

Minor modes in effect:
  shell-dirtrack-mode: t
  gpm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(completion dos-w32 find-cmd find-dired whitespace cl-print debug
backtrace rmail-spam-filter rmailedit rmailsort undigest tramp-gvfs
zeroconf tramp-cache bug-reference shortdoc help-fns radix-tree
descr-text help-at-pt ispell unrmail time-stamp texinfo url-http
url-auth url-gw nsm tramp tramp-loaddefs trampver tramp-integration
tramp-compat ls-lisp arc-mode archive-mode srecode/srt-mode
semantic/analyze semantic/sort semantic/scope semantic/analyze/fcn
semantic/db semantic/format srecode/template srecode/srt-wy
semantic/wisent semantic/wisent/wisent semantic/ctxt srecode/ctxt
semantic/tag-ls semantic/find srecode/compile srecode/dictionary
srecode/fields srecode/table srecode eieio-base semantic/util-modes
semantic/util semantic semantic/tag semantic/lex semantic/fw
mode-local cedet pcmpl-unix rect compare-w novice kmacro etags
fileloop xref project quail mail-extr pp shadow emacsbug smerge-mode
diff log-edit pcvs-util add-log org-element avl-tree generator ol-eww
ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect gnus-search eieio-opt
speedbar ezimage dframe gnus-art mm-uu mml2015 mm-view mml-smime smime
dig gnus-sum gnus-group gnus-undo gnus-start gnus-dbus dbus gnus-cloud
nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int
gnus-range gnus-win ol-docview doc-view jka-compr image-mode exif
ol-bibtex bibtex ol-bbdb ol-w3m org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
org-list org-faces org-entities noutline outline org-version
ob-emacs-lisp ob-core ob-eval org-table ol org-keys org-compat
org-macs org-loaddefs format-spec find-func cal-menu calendar
cal-loaddefs cl-extra parse-time iso8601 mhtml-mode css-mode smie eww
xdg url-queue mm-url gnus nnheader wid-edit color js imenu cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs sgml-mode help-mode mule-util shell pcomplete
thingatpt files-x grep compile comint ansi-color ring misearch
multi-isearch epa-mail rmailkwd rmailsum vc-mtn vc-hg vc-git diff-mode
easy-mmode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc vc-dispatcher
shr kinsoku svg xml dom rmailout dabbrev mailalias sendmail qp rmailmm
message rmc puny rfc822 mml mml-sec epa epg epg-config gnus-util
text-property-search time-date mm-decode mm-bodies mm-encode
mailabbrev gmm-utils mailheader mail-parse rfc2231 rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils dired-aux dired dired-loaddefs t-mouse term/linux view
derived paren cus-start cus-load advice finder-inf package easymenu
browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode
elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet
lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese composite charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 1190950 169604)
 (symbols 48 45903 20)
 (strings 32 226599 25400)
 (string-bytes 1 5447911)
 (vectors 16 76348)
 (vector-slots 8 2114801 141195)
 (floats 8 441 551)
 (intervals 56 152476 898)
 (buffers 984 297))
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sat, 26 Dec 2020 05:18:53 -0500
> 
> Trying to debug a bug in compile-goto-error on Rmail files,
> I typed C-h f compilation--message->loc RET
> then clicked on the file name -- which is supposed to find the definition
> of that function.
> 
> It did not find the definition; instead it said,
> 
>     Unable to find location in file
> 
> I also tried M-x find-function RET compilation--message->loc RET.
> It found a call to compilation--message->loc, not the definition.
> 
> I searched for that name in the file and did not find a definition.
> I will try grepping for it.

It's a general problem with uses of cl-defstruct and similar
constructs: they generate functions and macros that the Help functions
are unable to find.  In this case, see this part of compile.el:

  (cl-defstruct (compilation--message
	      (:constructor nil)
	      (:copier nil)
	      ;; (:type list)                ;Old representation.
	      (:constructor compilation--make-message (loc type end-loc rule))
	      (:conc-name compilation--message->))
    loc type end-loc rule)

bug#45443: 28.0.50; Can't find definition of compilation--message->loc

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It's a general problem with uses of cl-defstruct and similar
  > constructs: they generate functions and macros that the Help functions
  > are unable to find.  In this case, see this part of compile.el:

  >   (cl-defstruct (compilation--message

What causes the problem?  There has to be a way to fix it.

Does the fact that the defining form name does not start with `def'
have anything to do with it?  We could call it `def-cl-struct'.

There is also this:

            (:conc-name compilation--message->))

Years have taught me that enabling a definition to be a little shorter
by making names by concatenation is a bad idea.  It makes the code
harder to understand because there are references to these names that
you can't find with simple searching.

Is there a way to rewrite that definition so it does not concatenate
names?

If there is none, can we create one?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)