bug#64735: 29.0.92; find invocations are ~15x slower because of ignores

[Spencer Baugh <sbaugh@janestreet.com>]

Dmitry Gutov <dmitry@gutov.dev> writes:
> On 20/07/2023 15:22, sbaugh@catern.com wrote:
>>> I'm not sure we should bother more than these two simple measures.
>> Unfortunately those two simple measures help rgrep but they don't help
>> project-find-regexp (and others project.el commands using
>> project--files-in-directory such as project-find-file), since those
>> project commands pull their ignores from the version control system
>> through vc (not grep-find-ignored-files), and then pass them to find.
>
> That's only a problem when the default file listing logic is used (and
> we usually delegate to something like 'git ls-files' instead, when the
> vc-aware backend is used).

Hm, yes, but things like C-u project-find-regexp will use the default
find-based file listing logic instead of git ls-files, as do a few other
things.

I wonder, could we just go ahead and make a vc function which is
list-files(GLOBS) and returns a list of files?  Both git and hg support
this.  Then we could have C-u project-find-regexp use that instead of
find, by taking the cross product of dirs-to-search and
file-name-patterns-to-search.  (And this would let me delete a big chunk
of my own project backend, so I'd be happy to implement it.)

Fundamentally it seems a little silly for project-ignores to ever be
used for a vc project; if the vcs gives us ignores, we can probably just
ask the vcs to list the files too, and it will have an efficient
implementation of that.

If we do that uniformly, then this find slowness would only affect
transient projects, and transient projects pull their ignores from
grep-find-ignored-files just like rgrep, so improvements will more
easily be applied to both.  (And maybe we could even get rid of
project-ignores entirely, then?)

> Anyway, some optimization could be useful there too. The extra
> difficulty, though, is that the entries in IGNORES already can come as
> wildcards. Can we merge several wildcards? Though I suppose if we use
> a regexp, we could construct an alternation anyway.
>
> Another question it would be helpful to check, is whether the
> different versions of 'find' out there work fine with -regex instead
> of -name, and don't get slowed down simply because of that
> feature. The old built-in 'find' on macOS, for example.