bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files

[Spencer Baugh <sbaugh@janestreet.com>]

Dmitry Gutov <dgutov@yandex.ru> writes:
> On 16/04/2023 00:56, sbaugh@catern.com wrote:
>
>>> Perhaps you could describe your case where you *did* see a significant
>>> improvement from this patch, and we can discuss the best steps to
>>> address that.
>> In short: I have a project.el backend for a large monorepo which has
>> a
>> project-files backend which returns only the subset of files which are
>> relevant to work happening in a given clone.  (Generally a user will
>> have many clones and be doing different work in each one.)  The
>> relevant-files subset is determined by integration with the build
>> system.
>> So running find returns a vast number of files and then searches
>> over
>> those, whereas running a search over project-files searches a much
>> smaller number of files.
>
> Neat.
>
>> Regarding your medium-term plans to improve project-files performance -
>> wildly guessing, but perhaps you have in mind a way to run a subprocess
>> that outputs the project-files list?  Let's call it
>> "project-files-process".  And then project-files-process could be piped
>> to grep instead, for maximum efficiency?  If that was the idea, then my
>> own backend could certainly have a project-files-process implementation
>> too, for maximum efficiency.
>
> That might be step number 3, although I'm not sure yet which kind of
> code will be required for the piping to be done efficiently enough.
>
> The other two things I was looking at are:
>
> - Use relative file names (less text to parse, memory to allocate, GC
>   to thrash). The awkward part is how to merge that with the idea that
>   project-files can include files from directories ("external
>   roots"). Split those off into a different method? Treat them as
>   separate projects to flat-map the lists of files at?
>
> - Add arguments to allow filtering the files using the underlying
>   tool. That can also result is much fewer files to parse in the
>   output under suitable circumstances (e.g. we'd be able to pass a
>  list of globs here).
>
> There is one implementation of the second item in the branch
> scratch/etags-regen.
>
> And both items need to be done carefully enough to maintain some
> backward compatibility.
>
> So unless you're in a hurry, give me a few weeks to get around to this.
>
> Further suggestions and patches are welcome, of course.

I'm in no hurry.  I will probably add this backend locally at my site in
the meantime.  We have no existing (non-trivial) xref-find-references
backend, so speeding this one up isn't too urgent (it's not competing
with anything), but definitely I am interested in project-files (and
project.el in general) speed improvements and will try to help out as it
becomes relevant.