all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Performance issue in project.el
       [not found] <20210627212123.xciejcjxrv2rxsra.ref@Ergus>
@ 2021-06-27 21:21 ` Ergus
  2021-06-28 11:42   ` Ergus
  0 siblings, 1 reply; 5+ messages in thread
From: Ergus @ 2021-06-27 21:21 UTC (permalink / raw)
  To: emacs-devel

Hi:

Using tramp I tried to use project.el with a command like
project-switch-to-buffer and it took like 10 minutes to complete.

I ran a profiler and I found that most of the time was taken by:

project-current called in a loop for all the opened buffers it calls
project--find-in-directory that calls
project-find-functions and there is going all the time.

I had global-tags-try-project-root (a function from an external package
tat call gnu global program... something like gtags). And I optimized
that function to use a cache and avoid calling the program for every
buffer creating a cache for it.

Something like:

```
(defvar global-tags--dbpath-cache (make-hash-table))

(cl-defun global-tags--get-dbpath (&optional (dir default-directory))
   "Filepath for database from DIR."
   (pcase (gethash dir global-tags--dbpath-cache)
     ('empty nil)
     ('nil
      (puthash dir 'empty global-tags--dbpath-cache)
      (when-let*
        === BLA ===
        (when (file-exists-p dbpath)
	 (puthash dir dbpath global-tags--dbpath-cache))))
     (code code)))
````

Now the time is half than before but still very slow to use the command
(around 3-5 minutes to complete) and running again the profiler I get this:

         5637  89% - command-execute
         5549  88%  - byte-code
         5549  88%   - project--read-project-buffer
         5549  88%    - let*
         5336  85%     - read-buffer
         5323  84%      - ivy-completing-read
         5323  84%       - ivy-read
         4941  78%        - ivy--reset-state
         4941  78%         - ivy--buffer-list
         4941  78%          - internal-complete-buffer
         4941  78%           - #<lambda -0x1a357caf01243d61>
         4941  78%            - and
         4941  78%             - equal
         4941  78%              - save-current-buffer
         4941  78%               - project-current
         4941  78%                - project--find-in-directory
         4548  72%                 - project-try-vc
         4537  72%                  - vc-responsible-backend
         4478  71%                   - #<compiled 0xd3f2e32af0966f7>
         4478  71%                    - vc-call-backend
         4478  71%                     - apply
         1470  23%                      + vc-svn-responsible-p
         1142  18%                      + vc-bzr-responsible-p
          970  15%                      + vc-hg-responsible-p
          390   6%                      + vc-git-responsible-p
          156   2%                      + vc-cvs-responsible-p
          126   2%                      + vc-rcs-responsible-p
          108   1%                      + vc-sccs-responsible-p
           98   1%                      + vc-src-responsible-p
           57   0%                   + tramp-file-name-handler
           11   0%                  + vc-file-getprop
          393   6%                 + global-tags-try-project-root
          375   5%        + read-from-minibuffer
           13   0%      + if
          213   3%     + project-current
           88   1%  + funcall-interactively
          572   9% + ...
           51   0% + timer-event-handler
            8   0% + redisplay_internal (C function)


As you can see most of the time is still taken by project-current and I
can't really understand why:

1) Are so many samples 4548 seems a very high number for only 25 opened
buffers.

2) why project-try-vc still takes so much...? Specially for unfrequent
vc systems in our days like svn or bzr that I am not using.

3) If vc-handled-backends has these value by default: '(RCS CVS SVN SCCS
SRC Bzr Git Hg Mtn) why is there some time taken by vc-hg-responsible-p
that is latter than git in the list?

In any case:

Maybe (I think I mentioned this before) `project.el` needs a sort of
cache to speedup some functions like `project-current` that are called
very frequently.

Best,
Ergus



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Performance issue in project.el
  2021-06-27 21:21 ` Performance issue in project.el Ergus
@ 2021-06-28 11:42   ` Ergus
  2021-06-28 12:14     ` Phil Sainty
  2021-06-28 14:10     ` Stefan Monnier
  0 siblings, 2 replies; 5+ messages in thread
From: Ergus @ 2021-06-28 11:42 UTC (permalink / raw)
  To: emacs-devel

On Sun, Jun 27, 2021 at 11:21:23PM +0200, Ergus wrote:
>Hi:
>
>Using tramp I tried to use project.el with a command like
>project-switch-to-buffer and it took like 10 minutes to complete.
>
>I ran a profiler and I found that most of the time was taken by:
>
>project-current called in a loop for all the opened buffers it calls
>project--find-in-directory that calls
>project-find-functions and there is going all the time.
>
>I had global-tags-try-project-root (a function from an external package
>tat call gnu global program... something like gtags). And I optimized
>that function to use a cache and avoid calling the program for every
>buffer creating a cache for it.
>
>Something like:
>
>```
>(defvar global-tags--dbpath-cache (make-hash-table))
>
>(cl-defun global-tags--get-dbpath (&optional (dir default-directory))
>  "Filepath for database from DIR."
>  (pcase (gethash dir global-tags--dbpath-cache)
>    ('empty nil)
>    ('nil
>     (puthash dir 'empty global-tags--dbpath-cache)
>     (when-let*
>       === BLA ===
>       (when (file-exists-p dbpath)
>	 (puthash dir dbpath global-tags--dbpath-cache))))
>    (code code)))
>````
>
>Now the time is half than before but still very slow to use the command
>(around 3-5 minutes to complete) and running again the profiler I get this:
>
>        5637  89% - command-execute
>        5549  88%  - byte-code
>        5549  88%   - project--read-project-buffer
>        5549  88%    - let*
>        5336  85%     - read-buffer
>        5323  84%      - ivy-completing-read
>        5323  84%       - ivy-read
>        4941  78%        - ivy--reset-state
>        4941  78%         - ivy--buffer-list
>        4941  78%          - internal-complete-buffer
>        4941  78%           - #<lambda -0x1a357caf01243d61>
>        4941  78%            - and
>        4941  78%             - equal
>        4941  78%              - save-current-buffer
>        4941  78%               - project-current
>        4941  78%                - project--find-in-directory
>        4548  72%                 - project-try-vc
>        4537  72%                  - vc-responsible-backend
>        4478  71%                   - #<compiled 0xd3f2e32af0966f7>
>        4478  71%                    - vc-call-backend
>        4478  71%                     - apply
>        1470  23%                      + vc-svn-responsible-p
>        1142  18%                      + vc-bzr-responsible-p
>         970  15%                      + vc-hg-responsible-p
>         390   6%                      + vc-git-responsible-p
>         156   2%                      + vc-cvs-responsible-p
>         126   2%                      + vc-rcs-responsible-p
>         108   1%                      + vc-sccs-responsible-p
>          98   1%                      + vc-src-responsible-p
>          57   0%                   + tramp-file-name-handler
>          11   0%                  + vc-file-getprop
>         393   6%                 + global-tags-try-project-root
>         375   5%        + read-from-minibuffer
>          13   0%      + if
>         213   3%     + project-current
>          88   1%  + funcall-interactively
>         572   9% + ...
>          51   0% + timer-event-handler
>           8   0% + redisplay_internal (C function)
>
>
>As you can see most of the time is still taken by project-current and I
>can't really understand why:
>
>1) Are so many samples 4548 seems a very high number for only 25 opened
>buffers.
>
>2) why project-try-vc still takes so much...? Specially for unfrequent
>vc systems in our days like svn or bzr that I am not using.
>
>3) If vc-handled-backends has these value by default: '(RCS CVS SVN SCCS
>SRC Bzr Git Hg Mtn) why is there some time taken by vc-hg-responsible-p
>that is latter than git in the list?
>
Just replying to myself about 3)

After reading the function vc-responsible-backend I can see that it uses
a mapcar to test the responsible; so all of them are always tested in
every single call every time.

As a workaround I removed all the uninteresting handlers from
vc-handled-backends and I get better times now, but IMHO it is still
very inefficient almost a minute for project-switch-to-buffer is
excessive.

VCS changing is not something that happens very often to require a check
of all the backends everytime, several times for every buffer in many
project.el functions right?

Should I open an issue for this?

Maybe it is possible to add a sort of optimization like the one
mentioned above to vc-responsible-backend?

Probably I am wrong, but: Do we really need to check all this everytime
we call project-current?  I mean, a file or default-dir changes project
so often that we need to check with all the backends constantly?

Either in the local system the performance penalty seems to be
significant.

>In any case:
>
>Maybe (I think I mentioned this before) `project.el` needs a sort of
>cache to speedup some functions like `project-current` that are called
>very frequently.
>
>Best,
>Ergus
>



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Performance issue in project.el
  2021-06-28 11:42   ` Ergus
@ 2021-06-28 12:14     ` Phil Sainty
  2021-06-28 12:44       ` Ergus
  2021-06-28 14:10     ` Stefan Monnier
  1 sibling, 1 reply; 5+ messages in thread
From: Phil Sainty @ 2021-06-28 12:14 UTC (permalink / raw)
  To: Ergus; +Cc: emacs-devel

On 2021-06-28 23:42, Ergus wrote:
> After reading the function vc-responsible-backend I can see that it 
> uses
> a mapcar to test the responsible; so all of them are always tested in
> every single call every time.

It didn't in Emacs 27, but see bug #42966 and commit 2697123933e3

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42966

commit 2697123933e3ac7ed4e21a6d12746a98ed7fa74a
Author: Lars Ingebrigtsen <larsi@gnus.org>
Date:   Mon Oct 26 21:10:36 2020 +0100

     Make vc-responsible-backend choose the most specific backend

     * lisp/vc/vc.el (vc-responsible-backend): Search through all the
     VC backends instead of the first one, and choose the one that's
     most specific (bug#42966).




^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Performance issue in project.el
  2021-06-28 12:14     ` Phil Sainty
@ 2021-06-28 12:44       ` Ergus
  0 siblings, 0 replies; 5+ messages in thread
From: Ergus @ 2021-06-28 12:44 UTC (permalink / raw)
  To: Phil Sainty; +Cc: emacs-devel

Reading the issue I see that there is already vc-file-prop-obarray.

Could vc-responsible-backend use that?

On Tue, Jun 29, 2021 at 12:14:57AM +1200, Phil Sainty wrote:
>On 2021-06-28 23:42, Ergus wrote:
>>After reading the function vc-responsible-backend I can see that it 
>>uses
>>a mapcar to test the responsible; so all of them are always tested in
>>every single call every time.
>
>It didn't in Emacs 27, but see bug #42966 and commit 2697123933e3
>
>https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42966
>
>commit 2697123933e3ac7ed4e21a6d12746a98ed7fa74a
>Author: Lars Ingebrigtsen <larsi@gnus.org>
>Date:   Mon Oct 26 21:10:36 2020 +0100
>
>    Make vc-responsible-backend choose the most specific backend
>
>    * lisp/vc/vc.el (vc-responsible-backend): Search through all the
>    VC backends instead of the first one, and choose the one that's
>    most specific (bug#42966).
>



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Performance issue in project.el
  2021-06-28 11:42   ` Ergus
  2021-06-28 12:14     ` Phil Sainty
@ 2021-06-28 14:10     ` Stefan Monnier
  1 sibling, 0 replies; 5+ messages in thread
From: Stefan Monnier @ 2021-06-28 14:10 UTC (permalink / raw)
  To: Ergus; +Cc: emacs-devel

> Should I open an issue for this?

Yes, unless you find there's one for it already of course,


        Stefan




^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-06-28 14:10 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <20210627212123.xciejcjxrv2rxsra.ref@Ergus>
2021-06-27 21:21 ` Performance issue in project.el Ergus
2021-06-28 11:42   ` Ergus
2021-06-28 12:14     ` Phil Sainty
2021-06-28 12:44       ` Ergus
2021-06-28 14:10     ` Stefan Monnier

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.