bug#22292: 25.0.50; xref-find-references doesn't find anything for Lisp symbols

bug#22292: 25.0.50; xref-find-references doesn't find anything for Lisp symbols

[Eli Zaretskii]

From: eliz@HOME-C4E4A596F7.i-did-not-set--mail-host-address--so-tickle-me
--text follows this line--
To reproduce:

  emacs -Q
  C-x C-f lisp/subr.el RET
  M-? executing-kbd-macro RET

The result is disappointing:

  No references found for: executing-kbd-macro

The equivalent command in a C buffer (for a C symbol) does work as
expected.

I'm guessing that something is missing in the Lisp xref back-end that
doesn't let us support this operation.  However, I don't think we can
ship Emacs 25.1 with such an omission.  Lisp is one of the 2 languages
in which Emacs is written, so we ought to support it well.

Can someone please implement the necessary functionality on the
emacs-25 branch?  Bonus points for doing that for a few more popular
languages for which we have support in lisp/progmodes/.

(This came up in the context of rewriting the "Tags Tables" section of
the user manual, where etags.el functionality now takes a back seat to
xref.el.  Part of that is documenting the xref-find-references
command.  But we cannot in good faith advertise a brand-new command
when its support for Lisp is so incomplete.  It doesn't look good.)

This bug will be a release blocker for v25.1.


In GNU Emacs 25.0.50.253 (i686-pc-mingw32)
 of 2016-01-02 built on HOME-C4E4A596F7
Repository revision: 964bea7da5283a6c1ca2d7c6ee8a9d07d4951e69
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
Configured using:
 'configure --prefix=/d/usr --enable-checking=yes,glyphs --with-wide-int
 --with-modules 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS MODULES

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Emacs-Lisp

Minor modes in effect:
  diff-auto-refine-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark saved where search started
user-error: No references found for: executing-kbd-macro
Scanning for dabbrevs...done
user-error: No dynamic expansion for ‘xref-find-ref’ found

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr dabbrev emacsbug message dired
format-spec rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util help-fns mail-prsvr mail-utils semantic/symref/grep
grep compile comint ansi-color semantic/symref semantic/util-modes
semantic/util semantic semantic/tag semantic/lex semantic/fw mode-local
find-func cedet vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs thingatpt
xref cl-seq project ring eieio eieio-core cl-macs character-fold
misearch multi-isearch vc vc-dispatcher vc-git diff-mode easy-mmode map
seq byte-opt gv bytecomp byte-compile cconv cl-extra help-mode easymenu
cl-loaddefs pcase cl-lib time-date mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp
disp-table w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 126058 6754)
 (symbols 56 23822 0)
 (miscs 48 45 110)
 (strings 16 36488 20905)
 (string-bytes 1 865921)
 (vectors 16 17696)
 (vector-slots 8 539561 4443)
 (floats 8 244 113)
 (intervals 40 648 200)
 (buffers 856 13))

bug#22292: 25.0.50; xref-find-references doesn't find anything for Lisp symbols

[Dmitry Gutov]

Hi Eli,

On 01/02/2016 03:23 PM, Eli Zaretskii wrote:
> To reproduce:
>
>    emacs -Q
>    C-x C-f lisp/subr.el RET
>    M-? executing-kbd-macro RET

This scenario works for me, and I get ~100 matches, with emacs-25 HEAD.

> The result is disappointing:
>
>    No references found for: executing-kbd-macro
>
> The equivalent command in a C buffer (for a C symbol) does work as
> expected.

I'd expect the problem to be OS-specific, but it working in C buffers is 
clearly odd. You should be able to edebug xref-collect-references to see 
what happens (*).

> I'm guessing that something is missing in the Lisp xref back-end that
> doesn't let us support this operation.  However, I don't think we can
> ship Emacs 25.1 with such an omission.  Lisp is one of the 2 languages
> in which Emacs is written, so we ought to support it well.

Agreed.

> Can someone please implement the necessary functionality on the
> emacs-25 branch?  Bonus points for doing that for a few more popular
> languages for which we have support in lisp/progmodes/.

The implementation is largely language-agnostic. Basically, it greps, 
and then checks the returned hits for matches for 
"\\_<executing-kbd-macro\\_>".

Which depends on each buffer's syntax table and 
syntax-propertize-function. We are going to encounter cases where the 
"current symbol" is detected wrong due to this heuristic, but the fixes 
will probably go into syntax tables and s-p-function values.

(*) However (!), for xref-find-references only, we delegate to 
semantic-symref-find-references-by-name. And it might call any of the 
registered tools (such as id-utils), if the respective database file is 
present.

Might it be that id-utils doesn't parse Elisp files? If you recall, I 
mentioned this as a source of a possible user confusion. I don't know 
how to make it friendlier.

> (This came up in the context of rewriting the "Tags Tables" section of
> the user manual, where etags.el functionality now takes a back seat to
> xref.el.  Part of that is documenting the xref-find-references
> command.

To be fair, we could leave xref-find-references unadvertised, because 
it's not replacing any of the existing etags commands. But yeah, it 
should work.

bug#22292: 25.0.50; xref-find-references doesn't find anything for Lisp symbols

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Jan 2016 16:05:33 +0200
> 
> >    emacs -Q
> >    C-x C-f lisp/subr.el RET
> >    M-? executing-kbd-macro RET
> 
> This scenario works for me, and I get ~100 matches, with emacs-25 HEAD.
> 
> > The result is disappointing:
> >
> >    No references found for: executing-kbd-macro
> >
> > The equivalent command in a C buffer (for a C symbol) does work as
> > expected.
> 
> I'd expect the problem to be OS-specific, but it working in C buffers is 
> clearly odd. You should be able to edebug xref-collect-references to see 
> what happens (*).

What does that use if an ID database is not available?  Grep?

Can you guide me through the call sequence?  Debugging xref and its
back-ends is a pain.

> (*) However (!), for xref-find-references only, we delegate to 
> semantic-symref-find-references-by-name. And it might call any of the 
> registered tools (such as id-utils), if the respective database file is 
> present.

I don't have an ID db for Emacs in this tree.

> Might it be that id-utils doesn't parse Elisp files?

They do, but that's not relevant in this case.

> To be fair, we could leave xref-find-references unadvertised, because 
> it's not replacing any of the existing etags commands.

I'm not replacing one command for another, I'm rewriting the entire
section.  It cannot be kept in its current form, since it assumes
everything is based on TAGS.

bug#22292: 25.0.50; xref-find-references doesn't find anything for Lisp symbols

[Dmitry Gutov]

On 01/02/2016 04:14 PM, Eli Zaretskii wrote:

> What does that use if an ID database is not available?  Grep?

Global, CScope, or, yes, Grep. I always end up using Grep.

> Can you guide me through the call sequence?  Debugging xref and its
> back-ends is a pain.

You don't need to worry about xref backends here, only about CEDET 
tools. Try debugging xref-collect-references.

It calls semantic-symref-find-references-by-name, which
calls semantic-symref-instantiate (which detects the tool used in the 
current directory), then delegates the search to the tool's logic via 
the generics semantic-symref-get-result and semantic-symref-perform-search.

The Grep implementation for that stuff lives in 
lisp/cedet/semantic/symref/grep.el.

> I'm not replacing one command for another, I'm rewriting the entire
> section.  It cannot be kept in its current form, since it assumes
> everything is based on TAGS.

That's great, but I think my point still stands: if worst comes to 
worst, we could leave this command unadvertised.

Not that I wouldn't try my best to fix any remaining small-to-medium 
problems, of course.

bug#22292: 25.0.50; xref-find-references doesn't find anything for Lisp symbols

[Eli Zaretskii]

> Cc: 22292@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Jan 2016 16:29:57 +0200
> 
> > Can you guide me through the call sequence?  Debugging xref and its
> > back-ends is a pain.
> 
> You don't need to worry about xref backends here, only about CEDET 
> tools. Try debugging xref-collect-references.

I did, but Edebug cannot step into cl-defmethod's, cl-defgeneric etc.
Any tricks up your sleeves to make that work?  I'd give half the
kingdom for being able to debug into those.

> It calls semantic-symref-find-references-by-name, which
> calls semantic-symref-instantiate (which detects the tool used in the 
> current directory), then delegates the search to the tool's logic via 
> the generics semantic-symref-get-result and semantic-symref-perform-search.
> 
> The Grep implementation for that stuff lives in 
> lisp/cedet/semantic/symref/grep.el.

Thanks, that was what I needed.

I fixed the problem.  I cannot believe we still have (had) such code
in our repository, but facts are stubborn.

So, one less bug that's blocking v25.1 ;-)

bug#22292: 25.0.50; xref-find-references doesn't find anything for Lisp symbols

[Dmitry Gutov]

On 01/02/2016 05:32 PM, Eli Zaretskii wrote:

> I did, but Edebug cannot step into cl-defmethod's, cl-defgeneric etc.
> Any tricks up your sleeves to make that work?  I'd give half the
> kingdom for being able to debug into those.

No, sorry. Mostly I use xref-find-references in this case. :) Or 
project-find-regexp.

But I _think_ edebug can be extended to support generic dispatch.

> I fixed the problem.  I cannot believe we still have (had) such code
> in our repository, but facts are stubborn.

Thank you. I didn't even think of semantic-symref-derive-find-filepatterns.

> So, one less bug that's blocking v25.1 ;-)

Yup!

bug#22292: 25.0.50; xref-find-references doesn't find anything for Lisp symbols

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Jan 2016 17:53:24 +0200
> 
> But I _think_ edebug can be extended to support generic dispatch.

That'd be awesome.