* Performance issue in project.el
[not found] <20210627212123.xciejcjxrv2rxsra.ref@Ergus>
@ 2021-06-27 21:21 ` Ergus
2021-06-28 11:42 ` Ergus
0 siblings, 1 reply; 5+ messages in thread
From: Ergus @ 2021-06-27 21:21 UTC (permalink / raw)
To: emacs-devel
Hi:
Using tramp I tried to use project.el with a command like
project-switch-to-buffer and it took like 10 minutes to complete.
I ran a profiler and I found that most of the time was taken by:
project-current called in a loop for all the opened buffers it calls
project--find-in-directory that calls
project-find-functions and there is going all the time.
I had global-tags-try-project-root (a function from an external package
tat call gnu global program... something like gtags). And I optimized
that function to use a cache and avoid calling the program for every
buffer creating a cache for it.
Something like:
```
(defvar global-tags--dbpath-cache (make-hash-table))
(cl-defun global-tags--get-dbpath (&optional (dir default-directory))
"Filepath for database from DIR."
(pcase (gethash dir global-tags--dbpath-cache)
('empty nil)
('nil
(puthash dir 'empty global-tags--dbpath-cache)
(when-let*
=== BLA ===
(when (file-exists-p dbpath)
(puthash dir dbpath global-tags--dbpath-cache))))
(code code)))
````
Now the time is half than before but still very slow to use the command
(around 3-5 minutes to complete) and running again the profiler I get this:
5637 89% - command-execute
5549 88% - byte-code
5549 88% - project--read-project-buffer
5549 88% - let*
5336 85% - read-buffer
5323 84% - ivy-completing-read
5323 84% - ivy-read
4941 78% - ivy--reset-state
4941 78% - ivy--buffer-list
4941 78% - internal-complete-buffer
4941 78% - #<lambda -0x1a357caf01243d61>
4941 78% - and
4941 78% - equal
4941 78% - save-current-buffer
4941 78% - project-current
4941 78% - project--find-in-directory
4548 72% - project-try-vc
4537 72% - vc-responsible-backend
4478 71% - #<compiled 0xd3f2e32af0966f7>
4478 71% - vc-call-backend
4478 71% - apply
1470 23% + vc-svn-responsible-p
1142 18% + vc-bzr-responsible-p
970 15% + vc-hg-responsible-p
390 6% + vc-git-responsible-p
156 2% + vc-cvs-responsible-p
126 2% + vc-rcs-responsible-p
108 1% + vc-sccs-responsible-p
98 1% + vc-src-responsible-p
57 0% + tramp-file-name-handler
11 0% + vc-file-getprop
393 6% + global-tags-try-project-root
375 5% + read-from-minibuffer
13 0% + if
213 3% + project-current
88 1% + funcall-interactively
572 9% + ...
51 0% + timer-event-handler
8 0% + redisplay_internal (C function)
As you can see most of the time is still taken by project-current and I
can't really understand why:
1) Are so many samples 4548 seems a very high number for only 25 opened
buffers.
2) why project-try-vc still takes so much...? Specially for unfrequent
vc systems in our days like svn or bzr that I am not using.
3) If vc-handled-backends has these value by default: '(RCS CVS SVN SCCS
SRC Bzr Git Hg Mtn) why is there some time taken by vc-hg-responsible-p
that is latter than git in the list?
In any case:
Maybe (I think I mentioned this before) `project.el` needs a sort of
cache to speedup some functions like `project-current` that are called
very frequently.
Best,
Ergus
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Performance issue in project.el
2021-06-27 21:21 ` Performance issue in project.el Ergus
@ 2021-06-28 11:42 ` Ergus
2021-06-28 12:14 ` Phil Sainty
2021-06-28 14:10 ` Stefan Monnier
0 siblings, 2 replies; 5+ messages in thread
From: Ergus @ 2021-06-28 11:42 UTC (permalink / raw)
To: emacs-devel
On Sun, Jun 27, 2021 at 11:21:23PM +0200, Ergus wrote:
>Hi:
>
>Using tramp I tried to use project.el with a command like
>project-switch-to-buffer and it took like 10 minutes to complete.
>
>I ran a profiler and I found that most of the time was taken by:
>
>project-current called in a loop for all the opened buffers it calls
>project--find-in-directory that calls
>project-find-functions and there is going all the time.
>
>I had global-tags-try-project-root (a function from an external package
>tat call gnu global program... something like gtags). And I optimized
>that function to use a cache and avoid calling the program for every
>buffer creating a cache for it.
>
>Something like:
>
>```
>(defvar global-tags--dbpath-cache (make-hash-table))
>
>(cl-defun global-tags--get-dbpath (&optional (dir default-directory))
> "Filepath for database from DIR."
> (pcase (gethash dir global-tags--dbpath-cache)
> ('empty nil)
> ('nil
> (puthash dir 'empty global-tags--dbpath-cache)
> (when-let*
> === BLA ===
> (when (file-exists-p dbpath)
> (puthash dir dbpath global-tags--dbpath-cache))))
> (code code)))
>````
>
>Now the time is half than before but still very slow to use the command
>(around 3-5 minutes to complete) and running again the profiler I get this:
>
> 5637 89% - command-execute
> 5549 88% - byte-code
> 5549 88% - project--read-project-buffer
> 5549 88% - let*
> 5336 85% - read-buffer
> 5323 84% - ivy-completing-read
> 5323 84% - ivy-read
> 4941 78% - ivy--reset-state
> 4941 78% - ivy--buffer-list
> 4941 78% - internal-complete-buffer
> 4941 78% - #<lambda -0x1a357caf01243d61>
> 4941 78% - and
> 4941 78% - equal
> 4941 78% - save-current-buffer
> 4941 78% - project-current
> 4941 78% - project--find-in-directory
> 4548 72% - project-try-vc
> 4537 72% - vc-responsible-backend
> 4478 71% - #<compiled 0xd3f2e32af0966f7>
> 4478 71% - vc-call-backend
> 4478 71% - apply
> 1470 23% + vc-svn-responsible-p
> 1142 18% + vc-bzr-responsible-p
> 970 15% + vc-hg-responsible-p
> 390 6% + vc-git-responsible-p
> 156 2% + vc-cvs-responsible-p
> 126 2% + vc-rcs-responsible-p
> 108 1% + vc-sccs-responsible-p
> 98 1% + vc-src-responsible-p
> 57 0% + tramp-file-name-handler
> 11 0% + vc-file-getprop
> 393 6% + global-tags-try-project-root
> 375 5% + read-from-minibuffer
> 13 0% + if
> 213 3% + project-current
> 88 1% + funcall-interactively
> 572 9% + ...
> 51 0% + timer-event-handler
> 8 0% + redisplay_internal (C function)
>
>
>As you can see most of the time is still taken by project-current and I
>can't really understand why:
>
>1) Are so many samples 4548 seems a very high number for only 25 opened
>buffers.
>
>2) why project-try-vc still takes so much...? Specially for unfrequent
>vc systems in our days like svn or bzr that I am not using.
>
>3) If vc-handled-backends has these value by default: '(RCS CVS SVN SCCS
>SRC Bzr Git Hg Mtn) why is there some time taken by vc-hg-responsible-p
>that is latter than git in the list?
>
Just replying to myself about 3)
After reading the function vc-responsible-backend I can see that it uses
a mapcar to test the responsible; so all of them are always tested in
every single call every time.
As a workaround I removed all the uninteresting handlers from
vc-handled-backends and I get better times now, but IMHO it is still
very inefficient almost a minute for project-switch-to-buffer is
excessive.
VCS changing is not something that happens very often to require a check
of all the backends everytime, several times for every buffer in many
project.el functions right?
Should I open an issue for this?
Maybe it is possible to add a sort of optimization like the one
mentioned above to vc-responsible-backend?
Probably I am wrong, but: Do we really need to check all this everytime
we call project-current? I mean, a file or default-dir changes project
so often that we need to check with all the backends constantly?
Either in the local system the performance penalty seems to be
significant.
>In any case:
>
>Maybe (I think I mentioned this before) `project.el` needs a sort of
>cache to speedup some functions like `project-current` that are called
>very frequently.
>
>Best,
>Ergus
>
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Performance issue in project.el
2021-06-28 11:42 ` Ergus
@ 2021-06-28 12:14 ` Phil Sainty
2021-06-28 12:44 ` Ergus
2021-06-28 14:10 ` Stefan Monnier
1 sibling, 1 reply; 5+ messages in thread
From: Phil Sainty @ 2021-06-28 12:14 UTC (permalink / raw)
To: Ergus; +Cc: emacs-devel
On 2021-06-28 23:42, Ergus wrote:
> After reading the function vc-responsible-backend I can see that it
> uses
> a mapcar to test the responsible; so all of them are always tested in
> every single call every time.
It didn't in Emacs 27, but see bug #42966 and commit 2697123933e3
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42966
commit 2697123933e3ac7ed4e21a6d12746a98ed7fa74a
Author: Lars Ingebrigtsen <larsi@gnus.org>
Date: Mon Oct 26 21:10:36 2020 +0100
Make vc-responsible-backend choose the most specific backend
* lisp/vc/vc.el (vc-responsible-backend): Search through all the
VC backends instead of the first one, and choose the one that's
most specific (bug#42966).
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Performance issue in project.el
2021-06-28 12:14 ` Phil Sainty
@ 2021-06-28 12:44 ` Ergus
0 siblings, 0 replies; 5+ messages in thread
From: Ergus @ 2021-06-28 12:44 UTC (permalink / raw)
To: Phil Sainty; +Cc: emacs-devel
Reading the issue I see that there is already vc-file-prop-obarray.
Could vc-responsible-backend use that?
On Tue, Jun 29, 2021 at 12:14:57AM +1200, Phil Sainty wrote:
>On 2021-06-28 23:42, Ergus wrote:
>>After reading the function vc-responsible-backend I can see that it
>>uses
>>a mapcar to test the responsible; so all of them are always tested in
>>every single call every time.
>
>It didn't in Emacs 27, but see bug #42966 and commit 2697123933e3
>
>https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42966
>
>commit 2697123933e3ac7ed4e21a6d12746a98ed7fa74a
>Author: Lars Ingebrigtsen <larsi@gnus.org>
>Date: Mon Oct 26 21:10:36 2020 +0100
>
> Make vc-responsible-backend choose the most specific backend
>
> * lisp/vc/vc.el (vc-responsible-backend): Search through all the
> VC backends instead of the first one, and choose the one that's
> most specific (bug#42966).
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Performance issue in project.el
2021-06-28 11:42 ` Ergus
2021-06-28 12:14 ` Phil Sainty
@ 2021-06-28 14:10 ` Stefan Monnier
1 sibling, 0 replies; 5+ messages in thread
From: Stefan Monnier @ 2021-06-28 14:10 UTC (permalink / raw)
To: Ergus; +Cc: emacs-devel
> Should I open an issue for this?
Yes, unless you find there's one for it already of course,
Stefan
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-06-28 14:10 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20210627212123.xciejcjxrv2rxsra.ref@Ergus>
2021-06-27 21:21 ` Performance issue in project.el Ergus
2021-06-28 11:42 ` Ergus
2021-06-28 12:14 ` Phil Sainty
2021-06-28 12:44 ` Ergus
2021-06-28 14:10 ` Stefan Monnier
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).