bug#59722: 30.0.50; project-find-regexp searches project-ignored files

[Dmitry Gutov <dgutov@yandex.ru>]

On 11/12/2022 14:38, Rudolf Adamkovič via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>>>     ((nil . ((project-ignores . ("apps/**/*.scm")))))
>>
>> project-vc-ignores, you mean.
> 
> Correct.  Dmitry, I apologize for causing confusion!

No problem. I just wanted to make double-sure that the typo is not the 
cause of the issue.

>>> Emacs ends up calling
>>>
>>>     git ls-files -z -c --exclude-standard \
>>>       --no-empty-directory -o -- :(exclude,glob,top)apps/**/*.scm
>>>
>>> [...]
>>
>> Huh, interesting. Could you try to show a similar command invocation
>> which would make Git output include a "logically empty" directory for
>> the Emacs repo? So that we have a common public project to compare.
> 
> I could not reproduce the problem with the Emacs source directory, but I
> did not give up!  And, after a whole lot of head-scratching, I found the
> root cause of the problem:
> 
> When `git ls-files' finds a symlink that points to a directory, it does
> *not* follow it.  Instead, it lists it in the output.  In my case, the
> symlink pointed to a directory with all of its files ignored.  However,
> and importantly, upon further testing, I found that ignores do *not*
> matter.

Okay, yes, that seems to be the case. And there seems to be no way to 
have 'git ls-files' include symlinks in the output.

I can suggest two options here, both non-ideal:

- Ignore the whole symlinked directories by adding ignore entries that 
match their symlink files, rather than directories. For example, by 
setting project-vc-ignores to '("apps/*") or '("apps/**"), or with 
individual entries for each of them. That is, if you want them ignored 
whole.

- Here's what seems to be a (Linux-only) way to have Git follow 
symlinks: https://github.com/Alcaro/GitBSLR (found at 
https://stackoverflow.com/a/49138555/615245).

Another automatic approach would be to add a new value in 
xref-search-program-alist that would first pipe through a 
non-symlink-ness checker. Or modify the existing values.

But I'm yet to write a suitable command or pipeline (help welcome!), so 
I'm not sure how practical that is, and what's the average overhead.

The obvious choice of (seq-filter #'file-regular-p (project-files 
(project-current))) creates a large overhead (takes about x3 time here 
as just listing the files). That too much, I think.