bug#23223: 25.0.92; Can xref-find-references be sped up?

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

With an ID database present in the Emacs top-level directory, do this:

 emacs -Q
 C-x C-f src/buffer.c RET
 C-u 1316 M-x goto-char RET

You should now see point on "current_buffer".  Now:

 M-x xref-find-references RET RET

When prompted for the project directory, remove the "src/" part, to
specify the root of the Emacs tree, and type RET.

This takes around 13 sec here, although the corresponding 'lid'
command:

  lid --result=grep -l -w current_buffer

takes only 0.07 sec.

The important part of the Lisp-level profile appears below.  It looks
like we are visiting each match of each file in the list of matches,
and that takes most of the time.  Can this be avoided somehow?  The
'lid' command already includes all the matches in Grep format, so why
are we visiting each match, when the information should be already
available?

  - command-execute                                                 833  92%
   - call-interactively                                             833  92%
    - funcall-interactively                                         830  92%
     - execute-extended-command                                     830  92%
      - command-execute                                             826  91%
       - call-interactively                                         815  90%
	- funcall-interactively                                     813  90%
	 - xref-find-references                                     813  90%
	  - xref--find-xrefs                                        813  90%
	   - xref-backend-references                                800  88%
	    - cl-mapcan                                             788  87%
	     - apply                                                788  87%
	      - cl-mapcar                                           788  87%
	       - mapcar                                             788  87%
		- #<compiled 0x200000000842dea0>                    788  87%
		 - xref-collect-references                          788  87%
		  - cl-mapcan                                       770  85%
		   - apply                                          770  85%
		    - cl-mapcar                                     770  85%
		     - mapcar                                       770  85%
		      - #<compiled 0x20000000084ea738>              770  85%
		       - xref--collect-matches                      728  80%
			- semantic-find-file-noselect               556  61%
			 - find-file-noselect                       556  61%
			  - find-file-noselect-1                    519  57%
			   - after-find-file                        503  55%
			    - normal-mode                           499  55%
			     - set-auto-mode                        493  54%
			      - set-auto-mode-0                     488  54%
			       - c-mode                             488  54%
				- c-common-init                     480  53%
				 - mapc                             473  52%
				  - #<compiled 0x2000000001c06b28>  472  52%
				   - c-neutralize-syntax-in-and-mark-CPP   466  51%
				    - c-extend-font-lock-region-for-macros 205  22%
				     - c-state-literal-at                  205  22%
				      - c-state-safe-place                 199  22%
				       - c-beginning-of-macro                3   0%
					  back-to-indentation                3   0%
					 c-syntactic-end-of-macro            1   0%
				      c-syntactic-end-of-macro               9   1%
				      c-neutralize-CPP-line                  7   0%
				    - c-beginning-of-macro                   4   0%
				       back-to-indentation                   2   0%
				    #<compiled 0x2000000001c06b00>           1   0%
				 - c-basic-common-init                       7   0%
				  - c-set-style                              4   0%
				   - mapc                                    4   0%
				    - #<compiled 0x2000000001ae61c0>         4   0%
				     - c-set-style-1                         2   0%
				      - mapcar                               2   0%
					 #<compiled 0x2000000001ae53a0>      1   0%
				- run-mode-hooks                             3   0%





In GNU Emacs 25.0.92.75 (i686-pc-mingw32)
 of 2016-04-04 built on HOME-C4E4A596F7
Repository revision: f501116ea896b20f195f5c841e8770d7fe0418b9
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
Configured using:
 'configure --prefix=/d/usr --enable-checking=yes,glyphs --with-wide-int
 --with-modules 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS MODULES

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode easymenu
cl-loaddefs pcase cl-lib mail-prsvr mail-utils time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 93253 6020)
 (symbols 56 20721 0)
 (miscs 48 36 107)
 (strings 16 17749 6443)
 (string-bytes 1 439871)
 (vectors 16 12404)
 (vector-slots 8 429720 3894)
 (floats 8 162 70)
 (intervals 40 261 118)
 (buffers 856 11))

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

On 04/05/2016 06:16 PM, Eli Zaretskii wrote:

> The important part of the Lisp-level profile appears below.  It looks
> like we are visiting each match of each file in the list of matches,
> and that takes most of the time.  Can this be avoided somehow?  The
> 'lid' command already includes all the matches in Grep format, so why
> are we visiting each match, when the information should be already
> available?

Huh, yes, that's the area of optimization I've forgotten about (mostly 
because of using plain Grep myself anyway). If id-utils (and some other 
tools we delegate to) know symbol boundaries, maybe we don't need to 
double-check.

However, Grep output, which 'lid --result=grep' also uses, only outputs 
the line number, but not the starting column.

Take this example:

I'm searching for 'buffer'. The last line of the output looks like this:

xmenu.c:832:      set_buffer_internal_1 (XBUFFER (buffer));

How do I find the appropriate 'buffer' match in this line?

Or can we ask 'lid' (and, ideally, Grep too) to include the column of a 
match in the output? And when there are several matches on the same 
line, output that line multiple times?

Grep has colorized output, so I should be able to work with that. But 
not 'lid', it seems.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

On 04/06/2016 03:37 AM, Dmitry Gutov wrote:

> If id-utils (and some other
> tools we delegate to) know symbol boundaries

Is that actually the case, though? At most, it's using some wonky 
heuristics.

For instance, if I run 'mkid' in the top directory of the Emacs sources, 
I can search for 'defmacro', and get a few lines with 'cl-defmacro' 
instead among the results. Or search for 'eval-when', and get some lines 
with 'eval-when-compile'. But not, curiously, 'cl-eval-when'.

id-utils version 4.6 here.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 6 Apr 2016 03:37:39 +0300
> 
> On 04/05/2016 06:16 PM, Eli Zaretskii wrote:
> 
> > The important part of the Lisp-level profile appears below.  It looks
> > like we are visiting each match of each file in the list of matches,
> > and that takes most of the time.  Can this be avoided somehow?  The
> > 'lid' command already includes all the matches in Grep format, so why
> > are we visiting each match, when the information should be already
> > available?
> 
> Huh, yes, that's the area of optimization I've forgotten about (mostly 
> because of using plain Grep myself anyway).

But using plain Grep has the same problem: Grep reports the hits, and
we then go ahead and visit each hit, right?

> If id-utils (and some other tools we delegate to) know symbol
> boundaries, maybe we don't need to double-check.

It should know.

> However, Grep output, which 'lid --result=grep' also uses, only outputs 
> the line number, but not the starting column.
> 
> Take this example:
> 
> I'm searching for 'buffer'. The last line of the output looks like this:
> 
> xmenu.c:832:      set_buffer_internal_1 (XBUFFER (buffer));
> 
> How do I find the appropriate 'buffer' match in this line?

You already have the code that looks for the match in each line.  What
I had in mind is looking in the output of Grep or lid, instead of
visiting the file and looking in that file's buffer.  Does this make
sense?

> Or can we ask 'lid' (and, ideally, Grep too) to include the column of a 
> match in the output?

No, I don't think such an option exists.  It could be a great
enhancement request, though ;-)

> Grep has colorized output, so I should be able to work with that. But 
> not 'lid', it seems.

Another enhancement to request, I guess.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 6 Apr 2016 18:09:55 +0300
> 
> On 04/06/2016 03:37 AM, Dmitry Gutov wrote:
> 
> > If id-utils (and some other
> > tools we delegate to) know symbol boundaries
> 
> Is that actually the case, though? At most, it's using some wonky 
> heuristics.
> 
> For instance, if I run 'mkid' in the top directory of the Emacs sources, 
> I can search for 'defmacro', and get a few lines with 'cl-defmacro' 
> instead among the results. Or search for 'eval-when', and get some lines 
> with 'eval-when-compile'. But not, curiously, 'cl-eval-when'.
> 
> id-utils version 4.6 here.

Sounds like a bug in reporting matches.  We invoke lid with the -w
switch, which should have reported only complete symbols, AFAIU
(although the documentation confusingly talks about "words").  Perhaps
report this to the developers.

Anyway, if you search in the hits instead of in the file's buffer,
this problem should not affect us, I think.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

On 04/06/2016 08:20 PM, Eli Zaretskii wrote:

> Sounds like a bug in reporting matches.

A bug in the matcher, maybe?

> We invoke lid with the -w
> switch, which should have reported only complete symbols, AFAIU
> (although the documentation confusingly talks about "words").  Perhaps
> report this to the developers.

I suspected the wording would mean that it just invokes Grep and uses 
the "boundary" anchors (\b), but no, doing a Grep search for 
"\beval-when\b" yields a bigger (and more consistent) set of matches.

So it's most likely a bug.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

On 04/06/2016 08:12 PM, Eli Zaretskii wrote:

> But using plain Grep has the same problem: Grep reports the hits, and
> we then go ahead and visit each hit, right?

Using Grep has the slowness problem, but it doesn't have this 
opportunity for optimization, since Grep definitely doesn't know what 
characters constitute a symbol in different languages. But your proposal 
may speed up Grep searches too, so that's a plus.

>> How do I find the appropriate 'buffer' match in this line?
>
> You already have the code that looks for the match in each line.  What
> I had in mind is looking in the output of Grep or lid, instead of
> visiting the file and looking in that file's buffer.  Does this make
> sense?

Yes, but the search for \_<eval-when\_> needs a properly set syntax 
table, syntax-propertize-function, and possibly other variables that the 
latter might depend on. Basically, it needs a buffer in the right major 
mode. And switching major modes is actually not cheap:

(benchmark 230 '(with-temp-buffer (c-mode)))

=> 1.20s

in my working Emacs instance (much faster in a pristine one, 
admittedly). But I'll have a try at doing this while reusing the 
temporary buffer.

Some major modes could also trip over incomplete source code, but we'll 
get to that when we see that. Syntax highlighting seems out of the 
question, though, with high likelihood of getting it wrong.

We also have the possibility of syntax-propertize-function not applying 
the correct syntax classes to the code fragment because it doesn't see 
the whole file, but I guess it's a small-ish price to pay for the 
performance improvement, and could be fixed on a case-by-case basis.

>> Or can we ask 'lid' (and, ideally, Grep too) to include the column of a
>> match in the output?
>
> No, I don't think such an option exists.  It could be a great
> enhancement request, though ;-)

I was hoping you could serve as a liaison in that, being the sole user 
of id-utils that I know of, so far.

But anyway, if there is no such feature now, we have to choose another 
solution in Emacs 25.1. Even if id-utils were to add this today, it 
would take time to trickle to the majority of our users.

The bar is actually higher: to make use of the enhancement, we'd need 
Global and CScope to support it, too, or at least to be able to detect 
the feature at runtime.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Cc: 23223@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 7 Apr 2016 02:11:39 +0300
> 
> On 04/06/2016 08:20 PM, Eli Zaretskii wrote:
> 
> > Sounds like a bug in reporting matches.
> 
> A bug in the matcher, maybe?

Could be, but less likely.  I looked at the Lisp scanner code and
didn't say anything that could explain this.  Moreover, the fid
command correctly shows both defmacro and cl-defmacro being present
in, say, lisp/ibuf-macs.el.

> > We invoke lid with the -w
> > switch, which should have reported only complete symbols, AFAIU
> > (although the documentation confusingly talks about "words").  Perhaps
> > report this to the developers.
> 
> I suspected the wording would mean that it just invokes Grep and uses 
> the "boundary" anchors (\b), but no, doing a Grep search for 
> "\beval-when\b" yields a bigger (and more consistent) set of matches.

No, lid doesn't invoke Grep, AFAIR.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Cc: 23223@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 7 Apr 2016 03:11:25 +0300
> 
>     We invoke lid with the -w
>     switch, which should have reported only complete symbols, AFAIU
>     (although the documentation confusingly talks about "words").  Perhaps
>     report this to the developers.
> 
> I suspected the wording would mean that it just invokes Grep and uses the "boundary" anchors (\b), but no, doing a Grep search for "\beval-when\b" yields a bigger (and more consistent) set of matches.
> 
> So it's most likely a bug.

Heh, I completely forgot how IDutils work.  The ID database doesn't
store the locations where the identifiers were found, all it stores is
the list of identifiers found in each file.  When lid is invoked, it
queries the database about the files that reference the identifier;
then it opens each file in the query results, and searches for the
identifier on each line of that file.  When lid is invoked to produce
a Grep-like output and with the -w option (which is what we do), it
searches each line for the moral equivalent of

  [^[:alnum:]_]IDENT[^[:alnum:]_]

where IDENT is the identifier we asked about.

So therefore, your observation:

> For instance, if I run 'mkid' in the top directory of the Emacs sources, I can search for 'defmacro', and get a few lines with 'cl-defmacro' instead among the results. Or search for 'eval-when', and get some lines with 'eval-when-compile'. But not, curiously, 'cl-eval-when'.

can now be perfectly explained, because none of the files we have in
Emacs that reference cl-eval-when also reference eval-when, but some
files that reference defmacro have matches for cl-defmacro, and some
files that reference eval-when have matches for eval-when-compile.

IOW, the ID database is used to minimize the number of files to
search, i.e. avoid searching files that cannot possibly result in a
match, that's all.

>         Or can we ask 'lid' (and, ideally, Grep too) to include the column of a
>         match in the output?
> 
>     No, I don't think such an option exists.  It could be a great
>     enhancement request, though ;-)
> 
> I was hoping you could serve as a liaison in that, being the sole user of id-utils that I know of, so far.

I could do that, but given the explanation above of what -w means, it
will hardly help us, right?

> But anyway, if there is no such feature now, we have to choose another solution in Emacs 25.1. Even if id-utils were to add this today, it would take time to trickle to the majority of our users.
> 
> The bar is actually higher: to make use of the enhancement, we'd need Global and CScope to support it, too, or at least to be able to detect the feature at runtime.

Right.  So I think we need to assume that the program we invoke to
bring the matches is not smart enough anyway, and Emacs will have to
search in the matches for those we want to present, filtering out the
false positives.  This assumption seems to fit well all the programs
that can support the queries required by xref-find-references.

I'm not sure I understand what you have in mind for that, though.  Are
you going to switch the major mode of the temporary buffer as
determined by the file name of each match?  Maybe we should take a
hint from the major mode of the buffer in which the command was
invoked?  Also, how about just using the syntax table of the major
mode, instead of actually turning the mode in the buffer -- could that
be significantly cheaper?

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

On 04/07/2016 06:03 PM, Eli Zaretskii wrote:

> Heh, I completely forgot how IDutils work.  The ID database doesn't
> store the locations where the identifiers were found, all it stores is
> the list of identifiers found in each file.  When lid is invoked, it
> queries the database about the files that reference the identifier;
> then it opens each file in the query results, and searches for the
> identifier on each line of that file.  When lid is invoked to produce
> a Grep-like output and with the -w option (which is what we do), it
> searches each line for the moral equivalent of
>
>   [^[:alnum:]_]IDENT[^[:alnum:]_]
>
> where IDENT is the identifier we asked about.

That explains everything, thanks.

>> I was hoping you could serve as a liaison in that, being the sole user of id-utils that I know of, so far.
>
> I could do that, but given the explanation above of what -w means, it
> will hardly help us, right?

It might if -w were to be improved, though. We can request for it to be 
smarter and do the post-filtering itself, using a language-aware 
scanner. Even if no other feature is added, the accuracy of results will 
improve for all users of id-utils, and Emacs will automatically end up 
having to do less work for this feature.

That's not pressing, though.

> Right.  So I think we need to assume that the program we invoke to
> bring the matches is not smart enough anyway, and Emacs will have to
> search in the matches for those we want to present, filtering out the
> false positives.  This assumption seems to fit well all the programs
> that can support the queries required by xref-find-references.

Probably.

> I'm not sure I understand what you have in mind for that, though.  Are
> you going to switch the major mode of the temporary buffer as
> determined by the file name of each match?

Yes. We can avoid switching if the major mode of the previous match was 
the same as the current one. Having too many major modes involved in one 
search is highly unlikely, so the switching overhead shouldn't be too 
much of a problem, actually.

> Maybe we should take a
> hint from the major mode of the buffer in which the command was
> invoked?

That could only work for the output of xref-find-references, but not 
project-find-regexp, which is also in need of speeding up.

Anyway, whether we "take a hint" or not, we're going to end up with 
imperfect results: if we don't visit the target buffers, we're going to 
have to ignore all the ways Emacs allows specifying the major mode, 
aside from auto-mode-alist ("mode: " specification, 
interpreter-mode-alist, magic-mode-alist, magic-fallback-mode-alist).

Hmm. Maybe we can still support most of those using 
(insert-file-contents "file-name" nil 0 200), at the cost of some extra 
overhead.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Cc: 23223@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 9 Apr 2016 06:12:29 +0300
> 
> >> I was hoping you could serve as a liaison in that, being the sole user of id-utils that I know of, so far.
> >
> > I could do that, but given the explanation above of what -w means, it
> > will hardly help us, right?
> 
> It might if -w were to be improved, though. We can request for it to be 
> smarter and do the post-filtering itself, using a language-aware 
> scanner. Even if no other feature is added, the accuracy of results will 
> improve for all users of id-utils, and Emacs will automatically end up 
> having to do less work for this feature.
> 
> That's not pressing, though.

I agree.

> > I'm not sure I understand what you have in mind for that, though.  Are
> > you going to switch the major mode of the temporary buffer as
> > determined by the file name of each match?
> 
> Yes. We can avoid switching if the major mode of the previous match was 
> the same as the current one. Having too many major modes involved in one 
> search is highly unlikely, so the switching overhead shouldn't be too 
> much of a problem, actually.

Would it help to only use the mode's syntax table, and avoid switching
on the major mode as a whole?

> Anyway, whether we "take a hint" or not, we're going to end up with 
> imperfect results: if we don't visit the target buffers, we're going to 
> have to ignore all the ways Emacs allows specifying the major mode, 
> aside from auto-mode-alist ("mode: " specification, 
> interpreter-mode-alist, magic-mode-alist, magic-fallback-mode-alist).

That problem is relevant for IDutils as well (the scanner is
determined by the file's extension only), so we already have a certain
(hopefully, small) number of misses and false positives.  I think this
cannot be entirely avoided.  So maybe we shouldn't try so hard
avoiding false positives.  E.g., the "M-x gid" command, which comes
with IDutils and is a trivial wrapper around lid invocation, simply
shows the output in a Grep-like buffer, through which you can step
with next-error.  It is lightning-fast: what takes 13 sec with
xref-find-references, takes less than 2 sec with "M-x gid".

> Hmm. Maybe we can still support most of those using 
> (insert-file-contents "file-name" nil 0 200), at the cost of some extra 
> overhead.

Perhaps use insert-file-contents-literally, as decoding could slow
down things considerably.  You are only looking for ASCII text.  Also
note that a mode can be specified in file-local variables at the
file's end, though.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2745 bytes --]

Please try the attached patch. It cuts the time to search for 
'current_buffer' from 4.5s to 0.75s here.

You can comment out different parts of xref--collect-matches, to test if 
there are any easy bottlenecks left. The only one I see is 
find-buffer-visiting; replacing it with get-file-buffer yields some 
extra boost, but it would come with corresponding downsides in those 
exact cases for which find-buffer-visiting was designed (although I can 
add another local variable instead that would cache the previous file 
name and buffer used for it; but that's scraping the bottom).

To measure, open src/xdisp.c and evaluate (benchmark 1 
'(xref-collect-references "current_buffer" default-directory)).

On 04/09/2016 10:25 AM, Eli Zaretskii wrote:

> Would it help to only use the mode's syntax table, and avoid switching
> on the major mode as a whole?

Not for performance anymore (see above). You can't always determine the 
syntax table from the major mode name (there is a convention, but it's 
not iron-clad), it might be in an autoloaded file, etc... And like I 
mentioned before, we also need syntax-propertize-function, and 
potentially any buffer-local variables that it could use.

> That problem is relevant for IDutils as well (the scanner is
> determined by the file's extension only), so we already have a certain
> (hopefully, small) number of misses and false positives.

That affects xref-find-references (as long as you use id-utils, which is 
not mandatory) but not project-find-regexp. The currently discussed 
tuneup should improve both.

> I think this
> cannot be entirely avoided.  So maybe we shouldn't try so hard
> avoiding false positives.

It seems we can't avoid ignoring the mode specification at the bottom, 
but that's about it. And nothing's stopping id-utils (or other tools) 
from using a better language detection scheme in the future.

> E.g., the "M-x gid" command, which comes
> with IDutils and is a trivial wrapper around lid invocation, simply
> shows the output in a Grep-like buffer, through which you can step
> with next-error.

Do you actually want xref-find-regexp, and not xref-find-references?

> It is lightning-fast: what takes 13 sec with
> xref-find-references, takes less than 2 sec with "M-x gid".

What's the new time you get from the former?

> Perhaps use insert-file-contents-literally, as decoding could slow
> down things considerably.

No significant difference here, on the given example.

By the way, the "insert-file-contents + set-auto-mode" dance comes with 
a new minor downside: extra chatter from the major modes. E.g. try 
project-file-regexp with "should have received a copy". We can avoid 
saving it to the message log, but it appears in the echo area either way.

[-- Attachment #2: xref-with-temp-buffer.diff --]
[-- Type: text/x-patch, Size: 11988 bytes --]

diff --git a/lisp/cedet/semantic/symref/cscope.el b/lisp/cedet/semantic/symref/cscope.el
index 4890b5b..3abd8b3 100644
--- a/lisp/cedet/semantic/symref/cscope.el
+++ b/lisp/cedet/semantic/symref/cscope.el
@@ -60,6 +60,9 @@ semantic-symref-tool-cscope
     (semantic-symref-parse-tool-output tool b)
     ))
 
+(defconst semantic-symref-cscope--line-re
+  "^\\([^ ]+\\) [^ ]+ \\([0-9]+\\) ")
+
 (cl-defmethod semantic-symref-parse-tool-output-one-line ((tool semantic-symref-tool-cscope))
   "Parse one line of grep output, and return it as a match list.
 Moves cursor to end of the match."
@@ -78,8 +81,13 @@ semantic-symref-tool-cscope
 	       ;; We have to return something at this point.
 	       subtxt)))
 	 )
-	(t
-	 (when (re-search-forward "^\\([^ ]+\\) [^ ]+ \\([0-9]+\\) " nil t)
+        ((eq (oref tool :resulttype) 'line-and-text)
+         (when (re-search-forward semantic-symref-cscope--line-re nil t)
+           (list (string-to-number (match-string 2))
+                 (expand-file-name (match-string 1))
+                 (buffer-substring-no-properties (point) (line-end-position)))))
+	(t ; :resulttype is 'line
+	 (when (re-search-forward semantic-symref-cscope--line-re nil t)
 	   (cons (string-to-number (match-string 2))
 		 (expand-file-name (match-string 1)))
 	   ))))
diff --git a/lisp/cedet/semantic/symref/global.el b/lisp/cedet/semantic/symref/global.el
index e4c114e..a33427e 100644
--- a/lisp/cedet/semantic/symref/global.el
+++ b/lisp/cedet/semantic/symref/global.el
@@ -49,6 +49,9 @@ semantic-symref-tool-global
     (semantic-symref-parse-tool-output tool b)
     ))
 
+(defconst semantic-symref-global--line-re
+  "^\\([^ ]+\\) +\\([0-9]+\\) \\([^ ]+\\) ")
+
 (cl-defmethod semantic-symref-parse-tool-output-one-line ((tool semantic-symref-tool-global))
   "Parse one line of grep output, and return it as a match list.
 Moves cursor to end of the match."
@@ -57,8 +60,13 @@ semantic-symref-tool-global
 	 ;; Search for files
 	 (when (re-search-forward "^\\([^\n]+\\)$" nil t)
 	   (match-string 1)))
+        ((eq (oref tool :resulttype) 'line-and-text)
+         (when (re-search-forward semantic-symref-global--line-re nil t)
+           (list (string-to-number (match-string 2))
+                 (match-string 3)
+                 (buffer-substring-no-properties (point) (line-end-position)))))
 	(t
-	 (when (re-search-forward "^\\([^ ]+\\) +\\([0-9]+\\) \\([^ ]+\\) " nil t)
+	 (when (re-search-forward semantic-symref-global--line-re nil t)
 	   (cons (string-to-number (match-string 2))
 		 (match-string 3))
 	   ))))
diff --git a/lisp/cedet/semantic/symref/grep.el b/lisp/cedet/semantic/symref/grep.el
index 5d1fea8..868e6c3 100644
--- a/lisp/cedet/semantic/symref/grep.el
+++ b/lisp/cedet/semantic/symref/grep.el
@@ -188,6 +188,9 @@ semantic-symref-grep-shell
     ;; Return the answer
     ans))
 
+(defconst semantic-symref-grep--line-re
+  "^\\(\\(?:[a-zA-Z]:\\)?[^:\n]+\\):\\([0-9]+\\):")
+
 (cl-defmethod semantic-symref-parse-tool-output-one-line ((tool semantic-symref-tool-grep))
   "Parse one line of grep output, and return it as a match list.
 Moves cursor to end of the match."
@@ -195,8 +198,13 @@ semantic-symref-grep-shell
 	 ;; Search for files
 	 (when (re-search-forward "^\\([^\n]+\\)$" nil t)
 	   (match-string 1)))
+        ((eq (oref tool :resulttype) 'line-and-text)
+         (when (re-search-forward semantic-symref-grep--line-re nil t)
+           (list (string-to-number (match-string 2))
+                 (match-string 1)
+                 (buffer-substring-no-properties (point) (line-end-position)))))
 	(t
-	 (when (re-search-forward  "^\\(\\(?:[a-zA-Z]:\\)?[^:\n]+\\):\\([0-9]+\\):" nil t)
+	 (when (re-search-forward semantic-symref-grep--line-re nil t)
 	   (cons (string-to-number (match-string 2))
 		 (match-string 1))
 	   ))))
diff --git a/lisp/cedet/semantic/symref/idutils.el b/lisp/cedet/semantic/symref/idutils.el
index 4127d7a..db3e9a0 100644
--- a/lisp/cedet/semantic/symref/idutils.el
+++ b/lisp/cedet/semantic/symref/idutils.el
@@ -49,6 +49,9 @@ semantic-symref-tool-idutils
     (semantic-symref-parse-tool-output tool b)
     ))
 
+(defconst semantic-symref-idutils--line-re
+  "^\\(\\(?:[a-zA-Z]:\\)?[^:\n]+\\):\\([0-9]+\\):")
+
 (cl-defmethod semantic-symref-parse-tool-output-one-line ((tool semantic-symref-tool-idutils))
   "Parse one line of grep output, and return it as a match list.
 Moves cursor to end of the match."
@@ -59,8 +62,13 @@ semantic-symref-tool-idutils
 	((eq (oref tool :searchtype) 'tagcompletions)
 	 (when (re-search-forward "^\\([^ ]+\\) " nil t)
 	   (match-string 1)))
-	(t
-	 (when (re-search-forward "^\\(\\(?:[a-zA-Z]:\\)?[^:\n]+\\):\\([0-9]+\\):" nil t)
+        ((eq (oref tool :resulttype) 'line-and-text)
+         (when (re-search-forward semantic-symref-idutils--line-re nil t)
+	   (list (string-to-number (match-string 2))
+                 (expand-file-name (match-string 1) default-directory)
+                 (buffer-substring-no-properties (point) (line-end-position)))))
+	(t ; resulttype is line
+	 (when (re-search-forward semantic-symref-idutils--line-re nil t)
 	   (cons (string-to-number (match-string 2))
 		 (expand-file-name (match-string 1) default-directory))
 	   ))))
diff --git a/lisp/progmodes/xref.el b/lisp/progmodes/xref.el
index feed0fb..7354f8b 100644
--- a/lisp/progmodes/xref.el
+++ b/lisp/progmodes/xref.el
@@ -859,19 +859,25 @@ xref-collect-references
   ;; to force the backend to use `default-directory'.
   (let* ((ede-minor-mode nil)
          (default-directory dir)
+         ;; FIXME: Remove CScope and Global from the recognized tools?
+         ;; The current implementations interpret the symbol search as
+         ;; "find all calls to the given function", but not function
+         ;; definition. And they return nothing when passed a variable
+         ;; name, even a global one.
          (semantic-symref-tool 'detect)
          (case-fold-search nil)
-         (res (semantic-symref-find-references-by-name symbol 'subdirs))
-         (hits (and res (oref res hit-lines)))
-         (orig-buffers (buffer-list)))
+         (inst (semantic-symref-instantiate :searchfor symbol
+                                            :searchtype 'symbol
+                                            :searchscope 'subdirs
+                                            :resulttype 'line-and-text))
+         (hits (semantic-symref-perform-search inst))
+         (tmp-buffer (generate-new-buffer " *xref-temp*")))
     (unwind-protect
         (cl-mapcan (lambda (hit) (xref--collect-matches
-                             hit (format "\\_<%s\\_>" (regexp-quote symbol))))
+                             hit (format "\\_<%s\\_>" (regexp-quote symbol))
+                             tmp-buffer))
                    hits)
-      ;; TODO: Implement "lightweight" buffer visiting, so that we
-      ;; don't have to kill them.
-      (mapc #'kill-buffer
-            (cl-set-difference (buffer-list) orig-buffers)))))
+      (kill-buffer tmp-buffer))))
 
 ;;;###autoload
 (defun xref-collect-matches (regexp files dir ignores)
@@ -890,34 +896,24 @@ xref-collect-matches
                                        files
                                        (expand-file-name dir)
                                        ignores))
-         (orig-buffers (buffer-list))
          (buf (get-buffer-create " *xref-grep*"))
          (grep-re (caar grep-regexp-alist))
-         (counter 0)
-         reporter
-         hits)
+         hits
+         (tmp-buffer (generate-new-buffer " *xref-temp*")))
     (with-current-buffer buf
       (erase-buffer)
       (call-process-shell-command command nil t)
       (goto-char (point-min))
       (while (re-search-forward grep-re nil t)
-        (push (cons (string-to-number (match-string 2))
-                    (match-string 1))
+        (push (list (string-to-number (match-string 2))
+                    (match-string 1)
+                    (buffer-substring-no-properties (point) (line-end-position)))
               hits)))
-    (setq reporter (make-progress-reporter
-                    (format "Collecting search results...")
-                    0 (length hits)))
     (unwind-protect
         (cl-mapcan (lambda (hit)
-                     (prog1
-                         (progress-reporter-update reporter counter)
-                       (cl-incf counter))
-                     (xref--collect-matches hit regexp))
+                     (xref--collect-matches hit regexp tmp-buffer))
                    (nreverse hits))
-      (progress-reporter-done reporter)
-      ;; TODO: Same as above.
-      (mapc #'kill-buffer
-            (cl-set-difference (buffer-list) orig-buffers)))))
+      (kill-buffer tmp-buffer))))
 
 (defun xref--rgrep-command (regexp files dir ignores)
   (require 'find-dired)      ; for `find-name-arg'
@@ -980,30 +976,49 @@ xref--regexp-to-extended
                (match-string 1 str)))))
    str t t))
 
-(defun xref--collect-matches (hit regexp)
-  (pcase-let* ((`(,line . ,file) hit)
-               (buf (or (find-buffer-visiting file)
-                        (semantic-find-file-noselect file))))
-    (with-current-buffer buf
-      (save-excursion
+(defvar xref--temp-buffer-file-name nil)
+
+(defun xref--collect-matches (hit regexp tmp-buffer)
+  (pcase-let* ((`(,line ,file ,text) hit)
+               (buf (find-buffer-visiting file)))
+    (if buf
+        (with-current-buffer buf
+          (save-excursion
+            (goto-char (point-min))
+            (forward-line (1- line))
+            (xref--collect-matches-1 regexp file line
+                                     (line-beginning-position)
+                                     (line-end-position))))
+      (with-current-buffer tmp-buffer
+        (erase-buffer)
+        (unless (equal file xref--temp-buffer-file-name)
+          (insert-file-contents file nil 0 200)
+          (setq-local delay-mode-hooks t)
+          (set-auto-mode t)
+          (setq-local xref--temp-buffer-file-name file)
+          (setq-local inhibit-read-only t)
+          (erase-buffer))
+        (insert text)
         (goto-char (point-min))
-        (forward-line (1- line))
-        (let ((line-end (line-end-position))
-              (line-beg (line-beginning-position))
-              matches)
-          (syntax-propertize line-end)
-          ;; FIXME: This results in several lines with the same
-          ;; summary. Solve with composite pattern?
-          (while (re-search-forward regexp line-end t)
-            (let* ((beg-column (- (match-beginning 0) line-beg))
-                   (end-column (- (match-end 0) line-beg))
-                   (loc (xref-make-file-location file line beg-column))
-                   (summary (buffer-substring line-beg line-end)))
-              (add-face-text-property beg-column end-column 'highlight
-                                      t summary)
-              (push (xref-make-match summary loc (- end-column beg-column))
-                    matches)))
-          (nreverse matches))))))
+        (xref--collect-matches-1 regexp file line
+                                 (point)
+                                 (point-max))))))
+
+(defun xref--collect-matches-1 (regexp file line line-beg line-end)
+  (let (matches)
+    (syntax-propertize line-end)
+    ;; FIXME: This results in several lines with the same
+    ;; summary. Solve with composite pattern?
+    (while (re-search-forward regexp line-end t)
+      (let* ((beg-column (- (match-beginning 0) line-beg))
+             (end-column (- (match-end 0) line-beg))
+             (loc (xref-make-file-location file line beg-column))
+             (summary (buffer-substring line-beg line-end)))
+        (add-face-text-property beg-column end-column 'highlight
+                                t summary)
+        (push (xref-make-match summary loc (- end-column beg-column))
+              matches)))
+    (nreverse matches)))
 
 (provide 'xref)
 

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Cc: 23223@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Apr 2016 02:27:34 +0300
> 
> Please try the attached patch. It cuts the time to search for 
> 'current_buffer' from 4.5s to 0.75s here.

Thanks, I see a similar speedup (from 13 sec to just 3).  Very
impressive.

Unfortunately, it seems to miss matches: out of 1127 matches of
current_buffer with the original version, the new one only shows 963.
It sounds like some conditions on what exactly is a symbol need
adjustment, because the new code finds this:

 122:    (buf == current_buffer ? BEGV			\

but not this:

  42: #define BEGV (current_buffer->begv)

IOW, if the symbol is followed by a "->", it is not recognized.

As an aside, if I invoke xref-find-references without an ID file,
which AFAIU means Emacs will invoke find+grep, I get this error:

  semantic-symref-derive-find-filepatterns: Customize ‘semantic-symref-filepattern-alist’ for lisp-interaction-mode

unless I invoke xref-find-references from a buffer in C mode.
Curiously, this doesn't happen when there's an ID file and IDutils is
invoked.  Is this expected?

> > E.g., the "M-x gid" command, which comes
> > with IDutils and is a trivial wrapper around lid invocation, simply
> > shows the output in a Grep-like buffer, through which you can step
> > with next-error.
> 
> Do you actually want xref-find-regexp, and not xref-find-references?

No, of course not. "gid" is just a short for "lid -R grep", so the
contents I get is the same as xref-find-references does, it's just
formatted differently.  And that is what I want: symbol hits, not just
regular expressions that ignore the source language syntax.1

> > It is lightning-fast: what takes 13 sec with
> > xref-find-references, takes less than 2 sec with "M-x gid".
> 
> What's the new time you get from the former?

3 sec (in an unoptimized build, I'd expect this to become 1 sec in an
optimized build).  So we are OK speed-wise, we just need to fix the
misses mentioned above.

> By the way, the "insert-file-contents + set-auto-mode" dance comes with 
> a new minor downside: extra chatter from the major modes. E.g. try 
> project-file-regexp with "should have received a copy".

I don't see this in xref-find-references.  Should I?

> We can avoid saving it to the message log, but it appears in the
> echo area either way.

Can't you bind inhibit-message to a non-nil value?

Thanks again for working on this.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2537 bytes --]

On 04/11/2016 06:56 PM, Eli Zaretskii wrote:

> Unfortunately, it seems to miss matches: out of 1127 matches of
> current_buffer with the original version, the new one only shows 963.
> It sounds like some conditions on what exactly is a symbol need
> adjustment,

Ooh, that's a great catch, thanks! Turns out, auto-mode-alist wasn't 
getting applied because buffer-file-name wasn't set. Fixing that added 
~25% performance hit, so I also added xref--find-buffer-visiting to 
cache the expensive lookup mentioned previously. See the new patch attached.

That seems to about exhaust the optimization opportunity here.

> As an aside, if I invoke xref-find-references without an ID file,
> which AFAIU means Emacs will invoke find+grep, I get this error:
>
>   semantic-symref-derive-find-filepatterns: Customize ‘semantic-symref-filepattern-alist’ for lisp-interaction-mode
>
> unless I invoke xref-find-references from a buffer in C mode.

...or the current major mode is one of the currently supported ones, via 
the above variable.

> Curiously, this doesn't happen when there's an ID file and IDutils is
> invoked.  Is this expected?

Yes. semantic-symref-filepattern-alist is defined in and used by 
semantic/symref/grep.el. We can add (lisp-interaction-mode "*.el") to it.

> "gid" is just a short for "lid -R grep", so the
> contents I get is the same as xref-find-references does, it's just
> formatted differently.

Not exactly: you get more false positives because it doesn't apply the 
language-aware filtering.

>> What's the new time you get from the former?
>
> 3 sec (in an unoptimized build, I'd expect this to become 1 sec in an
> optimized build).  So we are OK speed-wise, we just need to fix the
> misses mentioned above.

Cool.

>> By the way, the "insert-file-contents + set-auto-mode" dance comes with
>> a new minor downside: extra chatter from the major modes. E.g. try
>> project-file-regexp with "should have received a copy".
>
> I don't see this in xref-find-references.  Should I?

Not at all. xref-find-references only searches in the files related to 
the current one by type (or in a predefined database like id-utils's one).

The chatter I got was from e.g. sh-mode (about the indentation variable 
being guessed), nxml-mode, and so on, with matches in ~4000 files.

>> We can avoid saving it to the message log, but it appears in the
>> echo area either way.
>
> Can't you bind inhibit-message to a non-nil value?

That works!

> Thanks again for working on this.

Thanks for the thoughtful bug report.

[-- Attachment #2: xref-with-temp-buffer.diff --]
[-- Type: text/x-patch, Size: 13517 bytes --]

diff --git a/lisp/cedet/semantic/symref/cscope.el b/lisp/cedet/semantic/symref/cscope.el
index 4890b5b..3abd8b3 100644
--- a/lisp/cedet/semantic/symref/cscope.el
+++ b/lisp/cedet/semantic/symref/cscope.el
@@ -60,6 +60,9 @@ semantic-symref-tool-cscope
     (semantic-symref-parse-tool-output tool b)
     ))
 
+(defconst semantic-symref-cscope--line-re
+  "^\\([^ ]+\\) [^ ]+ \\([0-9]+\\) ")
+
 (cl-defmethod semantic-symref-parse-tool-output-one-line ((tool semantic-symref-tool-cscope))
   "Parse one line of grep output, and return it as a match list.
 Moves cursor to end of the match."
@@ -78,8 +81,13 @@ semantic-symref-tool-cscope
 	       ;; We have to return something at this point.
 	       subtxt)))
 	 )
-	(t
-	 (when (re-search-forward "^\\([^ ]+\\) [^ ]+ \\([0-9]+\\) " nil t)
+        ((eq (oref tool :resulttype) 'line-and-text)
+         (when (re-search-forward semantic-symref-cscope--line-re nil t)
+           (list (string-to-number (match-string 2))
+                 (expand-file-name (match-string 1))
+                 (buffer-substring-no-properties (point) (line-end-position)))))
+	(t ; :resulttype is 'line
+	 (when (re-search-forward semantic-symref-cscope--line-re nil t)
 	   (cons (string-to-number (match-string 2))
 		 (expand-file-name (match-string 1)))
 	   ))))
diff --git a/lisp/cedet/semantic/symref/global.el b/lisp/cedet/semantic/symref/global.el
index e4c114e..a33427e 100644
--- a/lisp/cedet/semantic/symref/global.el
+++ b/lisp/cedet/semantic/symref/global.el
@@ -49,6 +49,9 @@ semantic-symref-tool-global
     (semantic-symref-parse-tool-output tool b)
     ))
 
+(defconst semantic-symref-global--line-re
+  "^\\([^ ]+\\) +\\([0-9]+\\) \\([^ ]+\\) ")
+
 (cl-defmethod semantic-symref-parse-tool-output-one-line ((tool semantic-symref-tool-global))
   "Parse one line of grep output, and return it as a match list.
 Moves cursor to end of the match."
@@ -57,8 +60,13 @@ semantic-symref-tool-global
 	 ;; Search for files
 	 (when (re-search-forward "^\\([^\n]+\\)$" nil t)
 	   (match-string 1)))
+        ((eq (oref tool :resulttype) 'line-and-text)
+         (when (re-search-forward semantic-symref-global--line-re nil t)
+           (list (string-to-number (match-string 2))
+                 (match-string 3)
+                 (buffer-substring-no-properties (point) (line-end-position)))))
 	(t
-	 (when (re-search-forward "^\\([^ ]+\\) +\\([0-9]+\\) \\([^ ]+\\) " nil t)
+	 (when (re-search-forward semantic-symref-global--line-re nil t)
 	   (cons (string-to-number (match-string 2))
 		 (match-string 3))
 	   ))))
diff --git a/lisp/cedet/semantic/symref/grep.el b/lisp/cedet/semantic/symref/grep.el
index 5d1fea8..868e6c3 100644
--- a/lisp/cedet/semantic/symref/grep.el
+++ b/lisp/cedet/semantic/symref/grep.el
@@ -188,6 +188,9 @@ semantic-symref-grep-shell
     ;; Return the answer
     ans))
 
+(defconst semantic-symref-grep--line-re
+  "^\\(\\(?:[a-zA-Z]:\\)?[^:\n]+\\):\\([0-9]+\\):")
+
 (cl-defmethod semantic-symref-parse-tool-output-one-line ((tool semantic-symref-tool-grep))
   "Parse one line of grep output, and return it as a match list.
 Moves cursor to end of the match."
@@ -195,8 +198,13 @@ semantic-symref-grep-shell
 	 ;; Search for files
 	 (when (re-search-forward "^\\([^\n]+\\)$" nil t)
 	   (match-string 1)))
+        ((eq (oref tool :resulttype) 'line-and-text)
+         (when (re-search-forward semantic-symref-grep--line-re nil t)
+           (list (string-to-number (match-string 2))
+                 (match-string 1)
+                 (buffer-substring-no-properties (point) (line-end-position)))))
 	(t
-	 (when (re-search-forward  "^\\(\\(?:[a-zA-Z]:\\)?[^:\n]+\\):\\([0-9]+\\):" nil t)
+	 (when (re-search-forward semantic-symref-grep--line-re nil t)
 	   (cons (string-to-number (match-string 2))
 		 (match-string 1))
 	   ))))
diff --git a/lisp/cedet/semantic/symref/idutils.el b/lisp/cedet/semantic/symref/idutils.el
index 4127d7a..db3e9a0 100644
--- a/lisp/cedet/semantic/symref/idutils.el
+++ b/lisp/cedet/semantic/symref/idutils.el
@@ -49,6 +49,9 @@ semantic-symref-tool-idutils
     (semantic-symref-parse-tool-output tool b)
     ))
 
+(defconst semantic-symref-idutils--line-re
+  "^\\(\\(?:[a-zA-Z]:\\)?[^:\n]+\\):\\([0-9]+\\):")
+
 (cl-defmethod semantic-symref-parse-tool-output-one-line ((tool semantic-symref-tool-idutils))
   "Parse one line of grep output, and return it as a match list.
 Moves cursor to end of the match."
@@ -59,8 +62,13 @@ semantic-symref-tool-idutils
 	((eq (oref tool :searchtype) 'tagcompletions)
 	 (when (re-search-forward "^\\([^ ]+\\) " nil t)
 	   (match-string 1)))
-	(t
-	 (when (re-search-forward "^\\(\\(?:[a-zA-Z]:\\)?[^:\n]+\\):\\([0-9]+\\):" nil t)
+        ((eq (oref tool :resulttype) 'line-and-text)
+         (when (re-search-forward semantic-symref-idutils--line-re nil t)
+	   (list (string-to-number (match-string 2))
+                 (expand-file-name (match-string 1) default-directory)
+                 (buffer-substring-no-properties (point) (line-end-position)))))
+	(t ; resulttype is line
+	 (when (re-search-forward semantic-symref-idutils--line-re nil t)
 	   (cons (string-to-number (match-string 2))
 		 (expand-file-name (match-string 1) default-directory))
 	   ))))
diff --git a/lisp/progmodes/xref.el b/lisp/progmodes/xref.el
index feed0fb..68021c5 100644
--- a/lisp/progmodes/xref.el
+++ b/lisp/progmodes/xref.el
@@ -839,16 +839,18 @@ xref-etags-mode
         (kill-local-variable 'xref-backend-functions))
     (setq-local xref-backend-functions xref-etags-mode--saved)))
 
-(declare-function semantic-symref-find-references-by-name "semantic/symref")
-(declare-function semantic-find-file-noselect "semantic/fw")
+(declare-function semantic-symref-instantiate "semantic/symref")
+(declare-function semantic-symref-perform-search "semantic/symref")
 (declare-function grep-expand-template "grep")
 (defvar ede-minor-mode) ;; ede.el
 
+(defvar xref--last-visiting-buffer nil)
+
 (defun xref-collect-references (symbol dir)
   "Collect references to SYMBOL inside DIR.
 This function uses the Semantic Symbol Reference API, see
-`semantic-symref-find-references-by-name' for details on which
-tools are used, and when."
+`semantic-symref-tool-alist' for details on which tools are used,
+and when."
   (cl-assert (directory-name-p dir))
   (require 'semantic/symref)
   (defvar semantic-symref-tool)
@@ -859,19 +861,26 @@ xref-collect-references
   ;; to force the backend to use `default-directory'.
   (let* ((ede-minor-mode nil)
          (default-directory dir)
+         ;; FIXME: Remove CScope and Global from the recognized tools?
+         ;; The current implementations interpret the symbol search as
+         ;; "find all calls to the given function", but not function
+         ;; definition. And they return nothing when passed a variable
+         ;; name, even a global one.
          (semantic-symref-tool 'detect)
          (case-fold-search nil)
-         (res (semantic-symref-find-references-by-name symbol 'subdirs))
-         (hits (and res (oref res hit-lines)))
-         (orig-buffers (buffer-list)))
+         (inst (semantic-symref-instantiate :searchfor symbol
+                                            :searchtype 'symbol
+                                            :searchscope 'subdirs
+                                            :resulttype 'line-and-text))
+         (hits (semantic-symref-perform-search inst))
+         xref--last-visiting-buffer
+         (tmp-buffer (generate-new-buffer " *xref-temp*")))
     (unwind-protect
         (cl-mapcan (lambda (hit) (xref--collect-matches
-                             hit (format "\\_<%s\\_>" (regexp-quote symbol))))
+                             hit (format "\\_<%s\\_>" (regexp-quote symbol))
+                             tmp-buffer))
                    hits)
-      ;; TODO: Implement "lightweight" buffer visiting, so that we
-      ;; don't have to kill them.
-      (mapc #'kill-buffer
-            (cl-set-difference (buffer-list) orig-buffers)))))
+      (kill-buffer tmp-buffer))))
 
 ;;;###autoload
 (defun xref-collect-matches (regexp files dir ignores)
@@ -890,34 +899,25 @@ xref-collect-matches
                                        files
                                        (expand-file-name dir)
                                        ignores))
-         (orig-buffers (buffer-list))
          (buf (get-buffer-create " *xref-grep*"))
          (grep-re (caar grep-regexp-alist))
-         (counter 0)
-         reporter
-         hits)
+         hits
+         xref--last-visiting-buffer
+         (tmp-buffer (generate-new-buffer " *xref-temp*")))
     (with-current-buffer buf
       (erase-buffer)
       (call-process-shell-command command nil t)
       (goto-char (point-min))
       (while (re-search-forward grep-re nil t)
-        (push (cons (string-to-number (match-string 2))
-                    (match-string 1))
+        (push (list (string-to-number (match-string 2))
+                    (match-string 1)
+                    (buffer-substring-no-properties (point) (line-end-position)))
               hits)))
-    (setq reporter (make-progress-reporter
-                    (format "Collecting search results...")
-                    0 (length hits)))
     (unwind-protect
         (cl-mapcan (lambda (hit)
-                     (prog1
-                         (progress-reporter-update reporter counter)
-                       (cl-incf counter))
-                     (xref--collect-matches hit regexp))
+                     (xref--collect-matches hit regexp tmp-buffer))
                    (nreverse hits))
-      (progress-reporter-done reporter)
-      ;; TODO: Same as above.
-      (mapc #'kill-buffer
-            (cl-set-difference (buffer-list) orig-buffers)))))
+      (kill-buffer tmp-buffer))))
 
 (defun xref--rgrep-command (regexp files dir ignores)
   (require 'find-dired)      ; for `find-name-arg'
@@ -980,30 +980,60 @@ xref--regexp-to-extended
                (match-string 1 str)))))
    str t t))
 
-(defun xref--collect-matches (hit regexp)
-  (pcase-let* ((`(,line . ,file) hit)
-               (buf (or (find-buffer-visiting file)
-                        (semantic-find-file-noselect file))))
-    (with-current-buffer buf
-      (save-excursion
+(defvar xref--temp-buffer-file-name nil)
+
+(defun xref--collect-matches (hit regexp tmp-buffer)
+  (pcase-let* ((`(,line ,file ,text) hit)
+               (buf (xref--find-buffer-visiting file)))
+    (if buf
+        (with-current-buffer buf
+          (save-excursion
+            (goto-char (point-min))
+            (forward-line (1- line))
+            (xref--collect-matches-1 regexp file line
+                                     (line-beginning-position)
+                                     (line-end-position))))
+      (with-current-buffer tmp-buffer
+        (erase-buffer)
+        (unless (equal file xref--temp-buffer-file-name)
+          (insert-file-contents file nil 0 200)
+          ;; Can't (setq-local delay-mode-hooks t) because of
+          ;; bug#23272, but the performance penalty seems minimal.
+          (let ((buffer-file-name file)
+                (inhibit-message t)
+                message-log-max)
+            (ignore-errors
+              (set-auto-mode t)))
+          (setq-local xref--temp-buffer-file-name file)
+          (setq-local inhibit-read-only t)
+          (erase-buffer))
+        (insert text)
         (goto-char (point-min))
-        (forward-line (1- line))
-        (let ((line-end (line-end-position))
-              (line-beg (line-beginning-position))
-              matches)
-          (syntax-propertize line-end)
-          ;; FIXME: This results in several lines with the same
-          ;; summary. Solve with composite pattern?
-          (while (re-search-forward regexp line-end t)
-            (let* ((beg-column (- (match-beginning 0) line-beg))
-                   (end-column (- (match-end 0) line-beg))
-                   (loc (xref-make-file-location file line beg-column))
-                   (summary (buffer-substring line-beg line-end)))
-              (add-face-text-property beg-column end-column 'highlight
-                                      t summary)
-              (push (xref-make-match summary loc (- end-column beg-column))
-                    matches)))
-          (nreverse matches))))))
+        (xref--collect-matches-1 regexp file line
+                                 (point)
+                                 (point-max))))))
+
+(defun xref--collect-matches-1 (regexp file line line-beg line-end)
+  (let (matches)
+    (syntax-propertize line-end)
+    ;; FIXME: This results in several lines with the same
+    ;; summary. Solve with composite pattern?
+    (while (re-search-forward regexp line-end t)
+      (let* ((beg-column (- (match-beginning 0) line-beg))
+             (end-column (- (match-end 0) line-beg))
+             (loc (xref-make-file-location file line beg-column))
+             (summary (buffer-substring line-beg line-end)))
+        (add-face-text-property beg-column end-column 'highlight
+                                t summary)
+        (push (xref-make-match summary loc (- end-column beg-column))
+              matches)))
+    (nreverse matches)))
+
+(defun xref--find-buffer-visiting (file)
+  (unless (equal (car xref--last-visiting-buffer) file)
+    (setq xref--last-visiting-buffer
+          (cons file (find-buffer-visiting file))))
+  (cdr xref--last-visiting-buffer))
 
 (provide 'xref)
 

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Cc: 23223@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 Apr 2016 02:25:54 +0300
> 
> > Unfortunately, it seems to miss matches: out of 1127 matches of
> > current_buffer with the original version, the new one only shows 963.
> > It sounds like some conditions on what exactly is a symbol need
> > adjustment,
> 
> Ooh, that's a great catch, thanks! Turns out, auto-mode-alist wasn't 
> getting applied because buffer-file-name wasn't set. Fixing that added 
> ~25% performance hit, so I also added xref--find-buffer-visiting to 
> cache the expensive lookup mentioned previously. See the new patch attached.
> 
> That seems to about exhaust the optimization opportunity here.

LGTM, please push.

Btw, I noticed another strange thing, but I was only able to reproduce
it with the current version, before your patch: the set of results
returned by xref-find-references, when it uses IDUtils, is not
entirely repeatable.  Sometimes, a few matches, sometimes a few dozen
of them, are missing.  I have no idea why; the lid command
consistently returns the same number of lines each time I invoke it.

> >   semantic-symref-derive-find-filepatterns: Customize ‘semantic-symref-filepattern-alist’ for lisp-interaction-mode
> >
> > unless I invoke xref-find-references from a buffer in C mode.
> 
> ...or the current major mode is one of the currently supported ones, via 
> the above variable.
> 
> > Curiously, this doesn't happen when there's an ID file and IDutils is
> > invoked.  Is this expected?
> 
> Yes. semantic-symref-filepattern-alist is defined in and used by 
> semantic/symref/grep.el. We can add (lisp-interaction-mode "*.el") to it.

I think we should do that, yes.

Thanks.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

Version: 25.1

On 04/12/2016 06:50 PM, Eli Zaretskii wrote:

> LGTM, please push.

Done, and closing.

> Btw, I noticed another strange thing, but I was only able to reproduce
> it with the current version, before your patch: the set of results
> returned by xref-find-references, when it uses IDUtils, is not
> entirely repeatable.  Sometimes, a few matches, sometimes a few dozen
> of them, are missing.  I have no idea why; the lid command
> consistently returns the same number of lines each time I invoke it.

That sounds troubling. Please report if you encounter this again, 
together with: do you get different results when immediately repeating 
the same search? Or do you search from different files, in different 
directories?

Getting different results between searches from c-mode and 
emacs-lisp-mode is to be expected, because the latter searches in all 
load-path as well (and it ends up using Grep in most of those 
directories anyway).

>semantic-symref-filepattern-alist is defined in and used by
>> semantic/symref/grep.el. We can add (lisp-interaction-mode "*.el") to it.
>
> I think we should do that, yes.

Also done (with a slightly longer value).

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Cc: 23223-done@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 Apr 2016 21:49:14 +0300
> 
> > Btw, I noticed another strange thing, but I was only able to reproduce
> > it with the current version, before your patch: the set of results
> > returned by xref-find-references, when it uses IDUtils, is not
> > entirely repeatable.  Sometimes, a few matches, sometimes a few dozen
> > of them, are missing.  I have no idea why; the lid command
> > consistently returns the same number of lines each time I invoke it.
> 
> That sounds troubling. Please report if you encounter this again, 

I just did, with the current emacs-25 branch.

> together with: do you get different results when immediately repeating 
> the same search? Or do you search from different files, in different 
> directories?

The same search, immediately repeated from the same directory.  More
accurately, the first time I invoke xref-find-references in the
top-level directory of the Emacs repository, I get 980 matches;
thereafter, all the subsequent searches from the same Emacs session
yield 1128 matches.  This is with the ID database present in the
top-level directory.

> Getting different results between searches from c-mode and 
> emacs-lisp-mode is to be expected, because the latter searches in all 
> load-path as well (and it ends up using Grep in most of those 
> directories anyway).

All the searches are done from *scratch* in "emacs -Q".

Let me know if I can provide more info.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

On 04/12/2016 10:16 PM, Eli Zaretskii wrote:

> The same search, immediately repeated from the same directory.  More
> accurately, the first time I invoke xref-find-references in the
> top-level directory of the Emacs repository, I get 980 matches;
> thereafter, all the subsequent searches from the same Emacs session
> yield 1128 matches.  This is with the ID database present in the
> top-level directory.

In the same scenario, I consistently get 1076 matches (on a GNU/Linux 
system): https://asciinema.org/a/69fdwo7os38fxu01e4fcwftwv

> All the searches are done from *scratch* in "emacs -Q".

The screencast above starts from Dired, but it's the same if I start 
from *scratch*.

> Let me know if I can provide more info.

Could you compare the lists of matches you get? E.g. C-x C-s on the xref 
buffer in both cases, and then diff the results.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Cc: 23223@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 Apr 2016 23:26:13 +0300
> 
> Could you compare the lists of matches you get? E.g. C-x C-s on the xref 
> buffer in both cases, and then diff the results.

Attached.

--- x00	2016-04-13 05:42:17.702125000 +0300
+++ x01	2016-04-13 05:42:35.092750000 +0300
@@ -692,14 +692,162 @@
 2174:     i = find_interval (buffer_intervals (current_buffer), pos);
 2465:   INTERVAL i = buffer_intervals (current_buffer);
 D:/gnu/git/emacs/branch/src/keyboard.c
- 736:       && current_buffer != XBUFFER (XWINDOW (selected_window)->contents))
-1410:       prev_buffer = current_buffer;
-1469:             buffer_before_last_command_or_undo = current_buffer;
-1516:       if (!NILP (BVAR (current_buffer, mark_active))
-1538: 		  && XMARKER (BVAR (current_buffer, mark))->buffer
-1553: 	      if (current_buffer != prev_buffer || MODIFF != prev_modiff)
-1562:       if (current_buffer == prev_buffer
+  736:       && current_buffer != XBUFFER (XWINDOW (selected_window)->contents))
+ 1410:       prev_buffer = current_buffer;
+ 1469:             buffer_before_last_command_or_undo = current_buffer;
+ 1516:       if (!NILP (BVAR (current_buffer, mark_active))
+ 1538: 		  && XMARKER (BVAR (current_buffer, mark))->buffer
+ 1553: 	      if (current_buffer != prev_buffer || MODIFF != prev_modiff)
+ 1562:       if (current_buffer == prev_buffer
+ 1563: 	  && XBUFFER (XWINDOW (selected_window)->contents) == current_buffer
+ 1633:   eassert (XBUFFER (XWINDOW (selected_window)->contents) == current_buffer);
+ 2833:       struct buffer *prev_buffer = current_buffer;
+ 2844:       if (current_buffer != prev_buffer)
+ 3905: 	  XSETBUFFER (obj, current_buffer);
+ 7392: 	if (tem = get_local_map (PT, current_buffer, Qkeymap), !NILP (tem))
+ 7399: 	maps[nmaps++] = get_local_map (PT, current_buffer, Qlocal_map);
+ 7945:       if (tem = get_local_map (PT, current_buffer, Qkeymap), !NILP (tem))
+ 7952:       maps[nmaps++] = get_local_map (PT, current_buffer, Qlocal_map);
+ 8808:    If FIX_CURRENT_BUFFER, we restore current_buffer
+ 8935:   starting_buffer = current_buffer;
+ 9151: 		      != current_buffer)
+ 9163: 	      && current_buffer != starting_buffer)
+ 9201: 		      != current_buffer))
+ 9252: 		      && XBUFFER (XWINDOW (window)->contents) != current_buffer)
+ 9642: 		      < XCHAR_TABLE (BVAR (current_buffer, downcase_table))->header.size)
+11759:    * confused because it doesn't realize that the current_buffer was
+D:/gnu/git/emacs/branch/src/keymap.c
+1486: 	  && XBUFFER (XWINDOW (window)->contents) != current_buffer)
+1515:       Lisp_Object local_map = get_local_map (pt, current_buffer, Qlocal_map);
+1517:       Lisp_Object keymap = get_local_map (pt, current_buffer, Qkeymap);
+1536: 					     current_buffer, Qlocal_map);
+1539: 					  current_buffer, Qkeymap);
+1677:   map = BVAR (current_buffer, keymap);
+1777:   bset_keymap (current_buffer, keymap);
+1787:   return BVAR (current_buffer, keymap);
+D:/gnu/git/emacs/branch/src/lread.c
+1470: 	  filename = Fexpand_file_name (filename, BVAR (current_buffer, directory));
+D:/gnu/git/emacs/branch/src/marker.c
+715:   for (tail = BUF_MARKERS (current_buffer); tail; tail = tail->next)
+750:       BUF_INC_POS (current_buffer, below_byte);
+D:/gnu/git/emacs/branch/src/minibuf.c
+422:   ambient_dir = BVAR (current_buffer, directory);
+538:       enable_multibyte = BVAR (current_buffer, enable_multibyte_characters);
+548:   bset_truncate_lines (current_buffer, Qnil);
+552:     bset_enable_multibyte_characters (current_buffer, enable_multibyte);
+557:      the current_buffer, so reset_buffer leaves buf's default
+563:     bset_directory (current_buffer, ambient_dir);
+571: 	    bset_directory (current_buffer,
+619:     if (!NILP (BVAR (current_buffer, enable_multibyte_characters))
+649:   bset_keymap (current_buffer, map);
+658:   bset_undo_list (current_buffer, Qnil);
+D:/gnu/git/emacs/branch/src/print.c
+ 91:    struct buffer *old = current_buffer;					\
+ 97:      = !NILP (BVAR (current_buffer, enable_multibyte_characters));	\
+102:        if (XBUFFER (printcharfun) != current_buffer)			\
+111:        if (XMARKER (printcharfun)->buffer != current_buffer)		\
+128:        if (NILP (BVAR (current_buffer, enable_multibyte_characters))	\
+131:        if (! NILP (BVAR (current_buffer, enable_multibyte_characters))	\
+158: 	   && NILP (BVAR (current_buffer, enable_multibyte_characters)))\
+302: 	    = !NILP (BVAR (current_buffer, enable_multibyte_characters));
+359: 	= !NILP (BVAR (current_buffer, enable_multibyte_characters));
+429: 		   : ! NILP (BVAR (current_buffer, enable_multibyte_characters))))
+543:   register struct buffer *old = current_buffer;
+551:   delete_all_overlays (current_buffer);
+552:   bset_directory (current_buffer, BVAR (old, directory));
+553:   bset_read_only (current_buffer, Qnil);
+554:   bset_filename (current_buffer, Qnil);
+555:   bset_undo_list (current_buffer, Qt);
+556:   eassert (current_buffer->overlays_before == NULL);
+557:   eassert (current_buffer->overlays_after == NULL);
+559:     (current_buffer, BVAR (&buffer_defaults, enable_multibyte_characters));
+563:   XSETBUFFER (buf, current_buffer);
+672:   struct buffer *previous = current_buffer;
+856:   struct buffer *old = current_buffer;
+D:/gnu/git/emacs/branch/src/process.c
+2194:     else if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
+3721:     else if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
+4604: 	      struct buffer *old_buffer = current_buffer;
+4612: 		  && (old_buffer != current_buffer
+4959: 	  struct buffer *old_buffer = current_buffer;
+4974: 	      && (old_buffer != current_buffer
+5377:   XSETBUFFER (obuffer, current_buffer);
+5378:   okeymap = BVAR (current_buffer, keymap);
+5462:       || ! EQ (current_buffer->keymap, okeymap))
+5497:       old_read_only = BVAR (current_buffer, read_only);
+5503:       bset_read_only (current_buffer, Qnil);
+5520:       if (NILP (BVAR (current_buffer, enable_multibyte_characters))
+5534: 	  && (b = XBUFFER (p->buffer), b != current_buffer))
+5563:       bset_read_only (current_buffer, old_read_only);
+5723: 	  struct buffer *cur = current_buffer;
+6559:   XSETBUFFER (obuffer, current_buffer);
+6560:   okeymap = BVAR (current_buffer, keymap);
+6607:       || ! EQ (current_buffer->keymap, okeymap))
+6720:       struct buffer *old = current_buffer;
+6730:       if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
+6747:       tem = BVAR (current_buffer, read_only);
+6748:       bset_read_only (current_buffer, Qnil);
+6754:       bset_read_only (current_buffer, tem);
+D:/gnu/git/emacs/branch/src/search.c
+ 147:   cp->syntax_table = cp->buf.used_syntax ? BVAR (current_buffer, syntax_table) : Qt;
+ 226: 	      || EQ (cp->syntax_table, BVAR (current_buffer, syntax_table)))
+ 275:   set_char_table_extras (BVAR (current_buffer, case_canon_table), 2,
+ 276: 			 BVAR (current_buffer, case_eqv_table));
+ 282: 			  (!NILP (BVAR (current_buffer, case_fold_search))
+ 283: 			   ? BVAR (current_buffer, case_canon_table) : Qnil),
+ 285: 			  !NILP (BVAR (current_buffer, enable_multibyte_characters)));
+ 333:     XSETBUFFER (last_thing_searched, current_buffer);
+ 391:   set_char_table_extras (BVAR (current_buffer, case_canon_table), 2,
+ 392: 			 BVAR (current_buffer, case_eqv_table));
+ 397: 			  (!NILP (BVAR (current_buffer, case_fold_search))
+ 398: 			   ? BVAR (current_buffer, case_canon_table) : Qnil),
+ 559:       multibyte = ! NILP (BVAR (current_buffer, enable_multibyte_characters));
+ 670:   newline_cache = newline_cache_on_off (current_buffer);
+ 671:   if (current_buffer->base_buffer)
+ 672:     cache_buffer = current_buffer->base_buffer;
+ 674:     cache_buffer = current_buffer;
+1055:   set_char_table_extras (BVAR (current_buffer, case_canon_table), 2,
+1056: 			 BVAR (current_buffer, case_eqv_table));
+1059: 		      (!NILP (BVAR (current_buffer, case_fold_search))
+1060: 		       ? BVAR (current_buffer, case_canon_table)
+1062: 		      (!NILP (BVAR (current_buffer, case_fold_search))
+1063: 		       ? BVAR (current_buffer, case_eqv_table)
+1189: 			      !NILP (BVAR (current_buffer, enable_multibyte_characters)));
+1243: 		  XSETBUFFER (last_thing_searched, current_buffer);
+1287: 		  XSETBUFFER (last_thing_searched, current_buffer);
+1312:       bool multibyte = !NILP (BVAR (current_buffer, enable_multibyte_characters));
+1528:   bool multibyte = ! NILP (BVAR (current_buffer, enable_multibyte_characters));
+1748:   bool multibyte = ! NILP (BVAR (current_buffer, enable_multibyte_characters));
+2159:   XSETBUFFER (last_thing_searched, current_buffer);
+2574:       bool buf_multibyte = !NILP (BVAR (current_buffer, enable_multibyte_characters));
+3220:     buf = current_buffer;
+3225:       old = current_buffer;
+D:/gnu/git/emacs/branch/src/syntax.c
+ 423: 	  gl_state.current_syntax_table = BVAR (current_buffer, syntax_table);
+ 563:   if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
+ 590:   if (current_buffer == find_start_buffer
+ 640:   find_start_buffer = current_buffer;
+ 981:   return BVAR (current_buffer, syntax_table);
+1026:   bset_syntax_table (current_buffer, table);
+1029:   SET_PER_BUFFER_VALUE_P (current_buffer, idx, 1);
+1233:     syntax_table = BVAR (current_buffer, syntax_table);
+1654:   multibyte = (!NILP (BVAR (current_buffer, enable_multibyte_characters))
+2137:   multibyte = (!NILP (BVAR (current_buffer, enable_multibyte_characters))
+2895: 		  if (! NILP (BVAR (current_buffer, enable_multibyte_characters)))
+D:/gnu/git/emacs/branch/src/term.c
+3692: 			 make_save_ptr_ptr (menu, current_buffer));
 D:/gnu/git/emacs/branch/src/textprop.c
+  83:   struct buffer *buf = XBUFFER (buffer), *old = current_buffer;
+  94:   bset_point_before_scroll (current_buffer, Qnil);
+ 525:     XSETBUFFER (object, current_buffer);
+ 566:     XSETBUFFER (object, current_buffer);
+ 610:     XSETBUFFER (object, current_buffer);
+ 622:       struct buffer *obuf = current_buffer;
+ 793:       if (BUFFERP (object) && current_buffer != XBUFFER (object))
+ 877:       if (BUFFERP (object) && current_buffer != XBUFFER (object))
+ 948:     XSETBUFFER (object, current_buffer);
+1015:     XSETBUFFER (object, current_buffer);
+1062:     XSETBUFFER (object, current_buffer);
 1112:     XSETBUFFER (object, current_buffer);
 1160:     XSETBUFFER (object, current_buffer);
 1360:     XSETBUFFER (object, current_buffer);

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Dmitry Gutov]

On 04/13/2016 05:44 AM, Eli Zaretskii wrote:

> Attached.

Thank you. Unfortunately, nothing jumps out. Could you try instrumenting 
a couple of functions, to see where the matches get lost?

First, semantic-symref-parse-tool-output, see whether the buffer 
contents are different between invocations (press `e' and evaluate 
(write-region (point-min) (point-max) "lid-output-x.txt")).

Next, xref--convert-hits. See that its gets the expected number of hits 
as input (plus maybe a few that we legitimately need to filter out).

If both are true, I guess there's nothing better to do than step through 
xref--convert-hits and see where it loses the matches. But I'm guessing 
the problem is somewhere before it.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Cc: 23223@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 13 Apr 2016 13:18:14 +0300
> 
> First, semantic-symref-parse-tool-output, see whether the buffer 
> contents are different between invocations

They are different: the first one has a long run of binary nulls right
in the middle of it.

So this is unrelated to xref-find-references, it's some basic problem
with buffer memory management.  Could be the same problem that causes
this:

  http://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00308.html

I will debug it, now that I have a reproducible recipe.

Thanks.

bug#23223: 25.0.92; Can xref-find-references be sped up?

[Eli Zaretskii]

> Date: Wed, 13 Apr 2016 18:16:53 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 23223@debbugs.gnu.org
> 
> > Cc: 23223@debbugs.gnu.org
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > Date: Wed, 13 Apr 2016 13:18:14 +0300
> > 
> > First, semantic-symref-parse-tool-output, see whether the buffer 
> > contents are different between invocations
> 
> They are different: the first one has a long run of binary nulls right
> in the middle of it.
> 
> So this is unrelated to xref-find-references, it's some basic problem
> with buffer memory management.  Could be the same problem that causes
> this:
> 
>   http://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00308.html
> 
> I will debug it, now that I have a reproducible recipe.

Done, see commit ab849b7 on the emacs-25 branch.