bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files

[Dmitry Gutov <dgutov@yandex.ru>]

Hi!

On 14/04/2023 18:37, Spencer Baugh wrote:
> When project-files is available, this is a much more efficient
> fallback than the current grep fallback.  Ultimately, this is
> motivated by making xref-find-references faster by default even in the
> absence of an index.

It's a clever enough idea, but unfortunately it doesn't look like the 
performance is always improved by this change.

E.g. I have this checkout of gecko-dev (a big project, just for testing: 
https://github.com/mozilla/gecko-dev) which contains different types of 
files: cpp, js, py.

If I do an xref-find-references search with the current code, it 
finishes in around ~0.8s. 'find' is not that slow, actually:

   time find . -type f -name "*.cpp" >/dev/null

reports just 400 ms here.

Whereas with your patch the search, depending on the language (cpp -- 
more files, py -- less files) can take 3 seconds and more.

Why? First of all, project-files returns all files (which are then all 
searched), whereas semantic-symref-filepattern-alist contains a mapping 
from modes to file globs, limiting both the scan and subsequent search 
to those.

Second -- using project-files means we're forced to round-trip the list 
of files names from the first project's stdout, to buffer, then to a 
list of Lisp strings, and then back to another buffer, to use as stdin. 
I have a couple of things planner in the medium term to improve that, 
but some overhead is probably unavoidable (unless we get some new 
primitive that would allow "piping" between process buffers).

Perhaps you could describe your case where you *did* see a significant 
improvement from this patch, and we can discuss the best steps to 
address that.

BTW, at first I figured you're using MacOS (which historically has 
bundled outdated versions of find and grep, with worse performance). But 
apparently not?