bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

On Tue, Jun 29, 2021 at 04:00:02PM +0300, Dmitry Gutov wrote:
>Hi!
>
>Thanks for the report, I'll follow up on this discussion some more 
>later, but some initial observations:
>
>On 29.06.2021 01:11, Ergus via Bug reports for GNU Emacs, the Swiss 
>army knife of text editors wrote:
>>As a workaround I removed all the uninteresting handlers from
>>vc-handled-backends and I get better times now, but IMHO it is still
>>very inefficient (almost a minute for project-switch-to-buffer is
>>excessive). And make it practically unusable.
>
>Could you evaluate (benchmark 1 '(project-current)) in one of your 
>buffers? That should give us an estimate how long it takes to find the 
>"current project" on your remote system.
>
Only with git and mercurial it takes:

Elapsed time: 3.018998s (0.109912s in 1 GCs)

With the entire list in vc-handled-backends

Elapsed time: 8.197923s (0.507396s in 6 GCs)

>If I'm right, project-switch-to-buffer should take 25 x (that time).
>
>If you indeed only have 25 buffers (including hidden ones).
>
>>In any case:
>>
>>Maybe (I think I mentioned this before) `project.el` needs a sort of
>>cache to speedup some functions like `project-current` that are called
>>very frequently inside the project.el code.
>
>The difficulty here is probably with the large latency to the remote 
>system. And our current approach calls the "find current project" 
>logic for each open buffer.
>
>Even if we add the "current project" cache, it will only take effect 
>at second and further invocations. Your first project-switch-to-buffer 
>call will still take 3-5 minutes, which is unacceptable.
>
>Please get back to us with the requested measurements, and perhaps 
>other observations (any research initiative is welcome), but if 
>finding the current vc root for each buffer is unavoidably slow, we'll 
>finally need to switch to a "project-contains-buffer-p generic" 
>approach, previously discussed in e.g. "Speed up project-kill-buffers" 
>thread on emacs-devel.

Another (and maybe even simpler) optimization, may be to consider that
all the buffers with the same default-directory should have the same vcs
and vc root (and probably belong to same project). (I think that all vcs
backends only do the search based on default-directory at the end.)

So if the association in the cache is for `default-directory` instead of
individual file names; then, less files will need to evaluate the search
of vc root, and vc-backend only 1/subdir will search the first time. It
is a very good trade of IMO.

This will reduce the time even the first time we iterate project-current
over all opened buffers if multiple files are in the same directory
(example: sources in src, includes in include and so on). And I think it
will cover 99% of normal use cases.

This won't affect nested projects, git-submodules or similar, because
those will be in different subdirs.