X-Debbugs-Cc: dmitry@gutov.dev

To see this in action, you can do the following, starting from "emacs 
-Q" inside of a Git repo that contains submodules:

   M-x trace-function RET project-files RET
   C-x p g some-string RET

If you look at the trace, you'll see that the files in your submodules 
are returned from 'project-files', but so is the submodule directory. 
That's not really correct, since 'project-files' is supposed to return 
*files*, not directories. (There are already workarounds for this in 
'project-search' and 'project-query-replace-regexp'.)

By default, this is just a theoretical problem, but if you customize 
'xref-search-program-alist' and 'xref-search-program' to include some 
other program, this can cause real issues. For example, I tried to add 
"ag" to this list[1], and unfortunately, it just doesn't work in this 
case. The results for submodules get duplicated, and there's no way I 
can see with ag to search only the specified *files*, ignoring any 
directories. (Looking at the definition for ripgrep, I'm guessing the 
"-g '!*/'" is the trick for that program, but nothing similar works for ag.)

Attached are two possible patches for this: a minimal version that just 
fixes 'project-find-regexp', and a maximal version that fixes this in 
general, and should theoretically speed up 'project-search' and 
'project-query-replace-regexp' since they no longer need to call 
'file-regular-p' on every file.

Do either of these patches look ok?

[1] This is a long story, simplified for this message since the gory 
details aren't especially relevant.