Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi <alexharsanyi@gmail.com>]

On 22 Jun 2018, at 4:57 pm, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>> Date: Fri, 22 Jun 2018 09:12:29 +0800
>> Cc: emacs-devel@gnu.org
>> 
>> I set `vc-command-messages`, which prints out the subprocesses run by
>> `vc-do-command` and also updated `vc-do-command` to show the time it took for
>> each process.  It turns out that Emacs 26.1 does run one extra command:
>> 
>> In Emacs 25.2, a "vc-dir" command runs the following:
>> 
>> Running git --no-pager update-index --refresh . in background... done
>> 0.48 seconds
>> Running git --no-pager diff-index --relative -z -M HEAD -- . in
>> background... done 0.42 seconds
>> Running git --no-pager ls-files -z -o --directory --no-empty-directory
>> --exclude-standard -- . in background... done 0.10 seconds
>> 
>> In Emacs 26.2, a "vc-dir" command runs the following:
>> 
>> Running git --no-pager update-index --refresh . in background... done
>> 0.46 seconds
>> Running git --no-pager diff-index --relative -z -M HEAD -- . in
>> background... done 0.46 seconds
>> Running git --no-pager ls-files -z -c -s -- . in background... done 4.49 seconds
>> Running git --no-pager ls-files -z -o --directory --no-empty-directory
>> --exclude-standard -- . in background... done 0.09 seconds
>> 
>> The extra command is "git --no-pager ls-files -z -c -s -- ." and it takes 4.5
>> seconds to run (I ran each test multiple times and the times are consistent).
>> However, when I run that command in a terminal, it finishes in less than a
>> second.
> 
> But the times you cited originally were much longer: 2 min.  What
> caused such a stark difference, from 2 min to just 4.5 sec?

The repository at my work place has more than 50000 files and vc-dir takes 2 minutes (measured using a stopwatch) for that repo. 

These tests I did at home against the Emacs repository which had approximately 3000 files. Originally I said this takes about 7 seconds, but that was “measured” by counting slowly in my head, these latest numbers actually measure the exact time. 

> 
>> An Emacs-Lisp program to run this script, but ignore all output, it takes 16
>> seconds, in both 25.2 and 26.1:
> 
> Once again, 16 sec is much shorter than 2 min you originally cited.

The ls-files command from for my work repo generates 6.5 Mb of data, my test program only generated 1 Mb of data.

> 
>> It seems reading output from a process in Emacs is somehow limited to 64 Kb /
>> second -- I verified this with procmon.  It looks like the behavior is the
>> same in 25.2 and 26.1, it is just that in vc-dir 25.2 does not run a git
>> command that produces a lot of output.
> 
> Does it help to set w32-pipe-buffer-size to some large value, like
> 16384, and/or decrease w32-pipe-read-delay?

I will try that tonight. Thanks for the suggestion. 

Alex

> 
> Thanks.