bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii <eliz@gnu.org>]

> Cc: 23223@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 7 Apr 2016 03:11:25 +0300
> 
>     We invoke lid with the -w
>     switch, which should have reported only complete symbols, AFAIU
>     (although the documentation confusingly talks about "words").  Perhaps
>     report this to the developers.
> 
> I suspected the wording would mean that it just invokes Grep and uses the "boundary" anchors (\b), but no, doing a Grep search for "\beval-when\b" yields a bigger (and more consistent) set of matches.
> 
> So it's most likely a bug.

Heh, I completely forgot how IDutils work.  The ID database doesn't
store the locations where the identifiers were found, all it stores is
the list of identifiers found in each file.  When lid is invoked, it
queries the database about the files that reference the identifier;
then it opens each file in the query results, and searches for the
identifier on each line of that file.  When lid is invoked to produce
a Grep-like output and with the -w option (which is what we do), it
searches each line for the moral equivalent of

  [^[:alnum:]_]IDENT[^[:alnum:]_]

where IDENT is the identifier we asked about.

So therefore, your observation:

> For instance, if I run 'mkid' in the top directory of the Emacs sources, I can search for 'defmacro', and get a few lines with 'cl-defmacro' instead among the results. Or search for 'eval-when', and get some lines with 'eval-when-compile'. But not, curiously, 'cl-eval-when'.

can now be perfectly explained, because none of the files we have in
Emacs that reference cl-eval-when also reference eval-when, but some
files that reference defmacro have matches for cl-defmacro, and some
files that reference eval-when have matches for eval-when-compile.

IOW, the ID database is used to minimize the number of files to
search, i.e. avoid searching files that cannot possibly result in a
match, that's all.

>         Or can we ask 'lid' (and, ideally, Grep too) to include the column of a
>         match in the output?
> 
>     No, I don't think such an option exists.  It could be a great
>     enhancement request, though ;-)
> 
> I was hoping you could serve as a liaison in that, being the sole user of id-utils that I know of, so far.

I could do that, but given the explanation above of what -w means, it
will hardly help us, right?

> But anyway, if there is no such feature now, we have to choose another solution in Emacs 25.1. Even if id-utils were to add this today, it would take time to trickle to the majority of our users.
> 
> The bar is actually higher: to make use of the enhancement, we'd need Global and CScope to support it, too, or at least to be able to detect the feature at runtime.

Right.  So I think we need to assume that the program we invoke to
bring the matches is not smart enough anyway, and Emacs will have to
search in the matches for those we want to present, filtering out the
false positives.  This assumption seems to fit well all the programs
that can support the queries required by xref-find-references.

I'm not sure I understand what you have in mind for that, though.  Are
you going to switch the major mode of the temporary buffer as
determined by the file name of each match?  Maybe we should take a
hint from the major mode of the buffer in which the command was
invoked?  Also, how about just using the syntax table of the major
mode, instead of actually turning the mode in the buffer -- could that
be significantly cheaper?