Speeding up vc-dired still more

Speeding up vc-dired still more

[Eric S. Raymond]

Dan Nicolaescu <dann@ics.uci.edu>:
> That will show  1000 calls to vc-bzr-registered, vc-git-registered,
> vc-arch-registered, vc-svn-registered etc etc.

Not any more.  I've added a check against
completion-ignored-extensions, encapsulated in a new function called
vc-dired-ignorable-p.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Rifles, muskets, long-bows and hand-grenades are inherently democratic
weapons.  A complex weapon makes the strong stronger, while a simple
weapon -- so long as there is no answer to it -- gives claws to the
weak.
        -- George Orwell, "You and the Atom Bomb", 1945

Re: Speeding up vc-dired still more

[Dan Nicolaescu]

"Eric S. Raymond" <esr@snark.thyrsus.com> writes:

  > Dan Nicolaescu <dann@ics.uci.edu>:
  > > That will show  1000 calls to vc-bzr-registered, vc-git-registered,
  > > vc-arch-registered, vc-svn-registered etc etc.
  > 
  > Not any more.  I've added a check against
  > completion-ignored-extensions, encapsulated in a new function called
  > vc-dired-ignorable-p.

Thanks, much better now.

There's still some more performance left to be squeezed out.

For the Linux kernel, with 5 files modified 
git ls-files | wc -l
22555

find . -type f 
shows a similar number.

The elp-results after doing an C-x v d in the kernel directory are:


Function Name                             Call Count  Elapsed Time  Average Time
========================================  ==========  ============  ============
vc-directory                              1           411.40415     411.40415
vc-dired-hook                             1           390.315596    390.315596
vc-backend                                22567       350.89581700  0.0155490679
vc-registered                             22553       348.03225700  0.0154317499
vc-call-backend                           135338      344.50350300  0.0025455046
vc-git-registered                         22553       315.02742200  0.0139683156
vc-git--out-ok                            22553       301.38389399  0.0133633615
vc-git--call                              22553       300.84999399  0.0133396884
^^^^^^^^^^^^
This seems to take most of the time. There are too many calls
vc-registered (which calls vc-git-registered, which in turn calls vc-git--call).

It seems that the vc-registered calls should not be necessary,
vc-git-dir-state should get the state for most of those files...


vc-find-root                              45111       22.290543000  0.0004941265
vc-default-registered                     45106       9.6466160000  0.0002138654
vc-git-root                               22556       8.4365439999  0.0003740266
vc-check-master-templates                 45106       8.0527110000  0.0001785285
vc-file-getprop                           112787      3.1722110000  2.812...e-05
vc-file-setprop                           45146       2.7922379999  6.184...e-05
vc-dired-ignorable-p                      22570       2.5674069999  0.0001137530
vc-possible-master                        135318      2.4839079999  1.835...e-05
vc-dired-purge                            1           1.728984      1.728984
vc-git-dir-state                          1           1.210754      1.210754
vc-state                                  22560       0.5940990000  2.633...e-05
vc-make-backend-sym                       45117       0.4681929999  1.037...e-05
vc-git-command                            1           0.144623      0.144623
vc-do-command                             1           0.144614      0.144614
vc-kill-buffer-hook                       22555       0.0699610000  3.101...e-06
vc-dired-mode                             1           0.060731      0.060731
vc-responsible-backend                    2           0.055134      0.027567
vc-find-backend-function                  11          0.0533559999  0.0048505454
vc-dired-reformat-line                    6           0.001267      0.0002111666
vc-git-responsible-p                      3           0.000387      0.000129
vc-git-dired-state-info                   6           0.000136      2.266...e-05
vc-delistify                              2           6e-06         3e-06
vc-exec-after                             1           6e-06         6e-06

Re: Speeding up vc-dired still more

[Dan Nicolaescu]

Dan Nicolaescu <dann@ics.uci.edu> writes:

  > "Eric S. Raymond" <esr@snark.thyrsus.com> writes:
  > 
  >   > Dan Nicolaescu <dann@ics.uci.edu>:
  >   > > That will show  1000 calls to vc-bzr-registered, vc-git-registered,
  >   > > vc-arch-registered, vc-svn-registered etc etc.
  >   > 
  >   > Not any more.  I've added a check against
  >   > completion-ignored-extensions, encapsulated in a new function called
  >   > vc-dired-ignorable-p.
  > 
  > Thanks, much better now.
  > 
  > There's still some more performance left to be squeezed out.
  > 
  > For the Linux kernel, with 5 files modified 
  > git ls-files | wc -l
  > 22555
  > 
  > find . -type f 
  > shows a similar number.
  > 
  > The elp-results after doing an C-x v d in the kernel directory are:
  > 
  > 
  > Function Name                             Call Count  Elapsed Time  Average Time
  > ========================================  ==========  ============  ============
  > vc-directory                              1           411.40415     411.40415
  > vc-dired-hook                             1           390.315596    390.315596
  > vc-backend                                22567       350.89581700  0.0155490679
  > vc-registered                             22553       348.03225700  0.0154317499
  > vc-call-backend                           135338      344.50350300  0.0025455046
  > vc-git-registered                         22553       315.02742200  0.0139683156
  > vc-git--out-ok                            22553       301.38389399  0.0133633615
  > vc-git--call                              22553       300.84999399  0.0133396884
  > ^^^^^^^^^^^^
  > This seems to take most of the time. There are too many calls
  > vc-registered (which calls vc-git-registered, which in turn calls vc-git--call).
  > 
  > It seems that the vc-registered calls should not be necessary,
  > vc-git-dir-state should get the state for most of those files...

All those vc-registered calls are generated by calling vc-backend in
vc-dired-hook:

         ;; ordinary file -- call the (possibly expensive) state query
         (t
           (let ((backend (vc-backend filename)))
               (cond
                    ;; Not registered
                         ((not backend)

Re: Speeding up vc-dired still more

[Eric S. Raymond]

Dan Nicolaescu <dann@ics.uci.edu>:
> All those vc-registered calls are generated by calling vc-backend in
> vc-dired-hook:
> 
>          ;; ordinary file -- call the (possibly expensive) state query
>          (t
>            (let ((backend (vc-backend filename)))
>                (cond
>                     ;; Not registered
>                          ((not backend)

I think this tells me that dir-state needs to set a new state value of
'unregistered when it sees status flag "?" (that's the Subversion and
Mercurial flag; there may be other local equivalents).  That way the
backend check wouldn't need to be done on on a per-file basis.

This change could be a bit tricky.  Right now a nil return from
vc-state means the file is unregistered.  I don't know how deeply that
assumption is wired into the code.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Speeding up vc-dired still more

[Alexandru Harsanyi]

On 27 Dec 2007, at 11:52 PM, Eric S. Raymond wrote:

> Dan Nicolaescu <dann@ics.uci.edu>:
>> That will show  1000 calls to vc-bzr-registered, vc-git-registered,
>> vc-arch-registered, vc-svn-registered etc etc.
>
> Not any more.  I've added a check against
> completion-ignored-extensions, encapsulated in a new function called
> vc-dired-ignorable-p.
>

Perhaps `vc-dired-ignorable-p' should be a backend function (vc- 
BACKEND-dired-ignorable-p).  I'm thinking that the CVS backend could  
look at .cvsignore for the list of files to ignore, the SVN backend  
at .svnignore, and so on.  This way, Emacs will ignore the same set  
of files as the VC backend would ignore when used in command line mode.

Alex.

Re: Speeding up vc-dired still more

[Eric S. Raymond]

Alexandru Harsanyi <harsanyi@mac.com>:
> Perhaps `vc-dired-ignorable-p' should be a backend function 
> (vc-BACKEND-dired-ignorable-p).  I'm thinking that the CVS backend could 
> look at .cvsignore for the list of files to ignore, the SVN backend at 
> .svnignore, and so on.  This way, Emacs will ignore the same set of files 
> as the VC backend would ignore when used in command line mode.

It shouldn't *be* a new backend function, but it should probably *call*
such a function.  The reason I say that is that there are some ignorable
files that don't fit in the .svnignore/.cvsignore model. 

As an example, right now I have in there logic to ignore Makefile if
it has a peer named Makefile.in or Makefile.am.

This is a good idea, but I have several higher-priority changes in my
queue.  Are you interested enough to do it?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Speeding up vc-dired still more

[Alexandru Harsanyi]

On 28 Dec 2007, at 6:25 PM, Eric S. Raymond wrote:

> Alexandru Harsanyi <harsanyi@mac.com>:
>> Perhaps `vc-dired-ignorable-p' should be a backend function
>> (vc-BACKEND-dired-ignorable-p).  I'm thinking that the CVS backend  
>> could
>> look at .cvsignore for the list of files to ignore, the SVN  
>> backend at
>> .svnignore, and so on.  This way, Emacs will ignore the same set  
>> of files
>> as the VC backend would ignore when used in command line mode.
>
> It shouldn't *be* a new backend function, but it should probably  
> *call*
> such a function.

This is what I intended to say too :-)

> The reason I say that is that there are some ignorable
> files that don't fit in the .svnignore/.cvsignore model.
>
> As an example, right now I have in there logic to ignore Makefile if
> it has a peer named Makefile.in or Makefile.am.

At least for the Emacs repository, .cvsignore lists Makefile so it is  
ignored by CVS.
A `vc-cvs-dired-ignorable-p' implementation  which looks only  
at .cvsignore will ignore the Makefile with no extra logic.

>
> This is a good idea, but I have several higher-priority changes in my
> queue.  Are you interested enough to do it?

I'm happy to make these changes, I will send you a patch for review.
>

Best Regards,
Alex.

Re: Speeding up vc-dired still more

[Eric S. Raymond]

Alexandru Harsanyi <harsanyi@mac.com>:
>> This is a good idea, but I have several higher-priority changes in my
>> queue.  Are you interested enough to do it?
>
> I'm happy to make these changes, I will send you a patch for review.

I now think this is actually only needed for CVS.  Subversion and
Mercurial will return an "I" in dirstate for ignored files -- it would
be faster to use that than to parse the ignore file explicitly. But
this will requires that I change (vc-state) to return an 'ignored
state.

Maybe you should hold off until I've reworked the repertoire of 
state properties.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>