Re: Performance issue in project.el

[Ergus <spacibba@aol.com>]

On Sun, Jun 27, 2021 at 11:21:23PM +0200, Ergus wrote:
>Hi:
>
>Using tramp I tried to use project.el with a command like
>project-switch-to-buffer and it took like 10 minutes to complete.
>
>I ran a profiler and I found that most of the time was taken by:
>
>project-current called in a loop for all the opened buffers it calls
>project--find-in-directory that calls
>project-find-functions and there is going all the time.
>
>I had global-tags-try-project-root (a function from an external package
>tat call gnu global program... something like gtags). And I optimized
>that function to use a cache and avoid calling the program for every
>buffer creating a cache for it.
>
>Something like:
>
>```
>(defvar global-tags--dbpath-cache (make-hash-table))
>
>(cl-defun global-tags--get-dbpath (&optional (dir default-directory))
>  "Filepath for database from DIR."
>  (pcase (gethash dir global-tags--dbpath-cache)
>    ('empty nil)
>    ('nil
>     (puthash dir 'empty global-tags--dbpath-cache)
>     (when-let*
>       === BLA ===
>       (when (file-exists-p dbpath)
>	 (puthash dir dbpath global-tags--dbpath-cache))))
>    (code code)))
>````
>
>Now the time is half than before but still very slow to use the command
>(around 3-5 minutes to complete) and running again the profiler I get this:
>
>        5637  89% - command-execute
>        5549  88%  - byte-code
>        5549  88%   - project--read-project-buffer
>        5549  88%    - let*
>        5336  85%     - read-buffer
>        5323  84%      - ivy-completing-read
>        5323  84%       - ivy-read
>        4941  78%        - ivy--reset-state
>        4941  78%         - ivy--buffer-list
>        4941  78%          - internal-complete-buffer
>        4941  78%           - #<lambda -0x1a357caf01243d61>
>        4941  78%            - and
>        4941  78%             - equal
>        4941  78%              - save-current-buffer
>        4941  78%               - project-current
>        4941  78%                - project--find-in-directory
>        4548  72%                 - project-try-vc
>        4537  72%                  - vc-responsible-backend
>        4478  71%                   - #<compiled 0xd3f2e32af0966f7>
>        4478  71%                    - vc-call-backend
>        4478  71%                     - apply
>        1470  23%                      + vc-svn-responsible-p
>        1142  18%                      + vc-bzr-responsible-p
>         970  15%                      + vc-hg-responsible-p
>         390   6%                      + vc-git-responsible-p
>         156   2%                      + vc-cvs-responsible-p
>         126   2%                      + vc-rcs-responsible-p
>         108   1%                      + vc-sccs-responsible-p
>          98   1%                      + vc-src-responsible-p
>          57   0%                   + tramp-file-name-handler
>          11   0%                  + vc-file-getprop
>         393   6%                 + global-tags-try-project-root
>         375   5%        + read-from-minibuffer
>          13   0%      + if
>         213   3%     + project-current
>          88   1%  + funcall-interactively
>         572   9% + ...
>          51   0% + timer-event-handler
>           8   0% + redisplay_internal (C function)
>
>
>As you can see most of the time is still taken by project-current and I
>can't really understand why:
>
>1) Are so many samples 4548 seems a very high number for only 25 opened
>buffers.
>
>2) why project-try-vc still takes so much...? Specially for unfrequent
>vc systems in our days like svn or bzr that I am not using.
>
>3) If vc-handled-backends has these value by default: '(RCS CVS SVN SCCS
>SRC Bzr Git Hg Mtn) why is there some time taken by vc-hg-responsible-p
>that is latter than git in the list?
>
Just replying to myself about 3)

After reading the function vc-responsible-backend I can see that it uses
a mapcar to test the responsible; so all of them are always tested in
every single call every time.

As a workaround I removed all the uninteresting handlers from
vc-handled-backends and I get better times now, but IMHO it is still
very inefficient almost a minute for project-switch-to-buffer is
excessive.

VCS changing is not something that happens very often to require a check
of all the backends everytime, several times for every buffer in many
project.el functions right?

Should I open an issue for this?

Maybe it is possible to add a sort of optimization like the one
mentioned above to vc-responsible-backend?

Probably I am wrong, but: Do we really need to check all this everytime
we call project-current?  I mean, a file or default-dir changes project
so often that we need to check with all the backends constantly?

Either in the local system the performance penalty seems to be
significant.

>In any case:
>
>Maybe (I think I mentioned this before) `project.el` needs a sort of
>cache to speedup some functions like `project-current` that are called
>very frequently.
>
>Best,
>Ergus
>