From: Alex Harsanyi <alexharsanyi@gmail.com>
To: Eli Zaretskii <eliz@gnu.org>
Cc: emacs-devel@gnu.org
Subject: Re: vc-dir operation is very slow on large git repositories in Emacs 26.1
Date: Mon, 25 Jun 2018 18:40:08 +0800 [thread overview]
Message-ID: <CAH-ciFrG2hfWwQeAPLnNP6QBkNw1wuquxgEvch=ZqfAY=hCARg@mail.gmail.com> (raw)
In-Reply-To: <83wouqptm6.fsf@gnu.org>
On Sat, Jun 23, 2018 at 2:24 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>> Date: Fri, 22 Jun 2018 20:46:59 +0800
>> Cc: emacs-devel@gnu.org
>>
>> > Does it help to set w32-pipe-buffer-size to some large value, like
>> > 16384, and/or decrease w32-pipe-read-delay?
>>
>> If I set w32-pipe-read-delay to 0 the speed improvement is
>> significant. On top of that, there is an additional speed
>> improvement if I set w32-pipe-buffer-size to 16384
>
> Can you show numbers for these speedups, and compare the times with
> Emacs 25. It would also be useful if you show times with different
> values of these, so that the optimal values could be chosen.
>
> I'd be also interested in the timings with the larger repository.
I tested various values for `w32-pipe-read-delay` on my work git repository.
The original timings that I made were against a work tree that had several
modified files and I used a stop watch to time the update. These tests were
run on a large (51678 tracked files) but clean working directory, and I
modified the source files to time the commands using `current-time`, so I got
more precise timings.
$ git ls-files . | wc -l
51678
$ git status
On branch ah_tmp
nothing to commit, working directory clean
Emacs 26.1 runs the following commands as part of a `vc-dir` command, or an
update of the `*vc-dir*` buffer (Emacs 25.2 runs the same commands, except it
does not run the third one)
* git --no-pager update-index --refresh .
* git --no-pager diff-index --relative -z -M HEAD -- .
* git --no-pager ls-files -z -c -s -- .
* git --no-pager ls-files -z -o --directory --no-empty-directory
--exclude-standard -- .
On a clean working directory, only "git --no-pager ls-files -z -c -s -- ."
produces any output, but produces 6.3 Mb of it. The command itself runs fast:
$ git --no-pager ls-files -z -c -s -- . | wc
0 184463 6689336
$ time git --no-pager ls-files -z -c -s -- . > /dev/null
real 0m0.619s
user 0m0.000s
sys 0m0.015s
I tried several values for `w32-pipe-read-delay` to see how long it takes to
run these commands in Emacs 26.1:
`w32-pipe-read-delay` is set to 50 milliseconds (default for Emacs 26.1)
Running git --no-pager update-index --refresh . in background...
done 0.26 seconds
Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.34 seconds
Running git --no-pager ls-files -z -c -s -- . in background...
done 83.70 seconds
Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.64 seconds
`w32-pipe-read-delay` set to 25 milliseconds
Running git --no-pager update-index --refresh . in background...
done 0.26 seconds
Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.34 seconds
Running git --no-pager ls-files -z -c -s -- . in background...
done 42.77 seconds
Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.67 seconds
`w32-pipe-read-delay` set to 12 milliseconds
Running git --no-pager update-index --refresh . in background...
done 0.27 seconds
Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.35 seconds
Running git --no-pager ls-files -z -c -s -- . in background...
done 21.57 seconds
Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.65 seconds
`w32-pipe-read-delay` set to 0 milliseconds (no delay)
Running git --no-pager update-index --refresh . in background...
done 0.31 seconds
Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.39 seconds
Running git --no-pager ls-files -z -c -s -- . in background...
done 1.07 seconds
Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.69 seconds
So, with a 50 millisecond delay, it takes 83.7 seconds to read the command
output and this is reduced to 1.07 seconds when the delay is set to 0.
Alex.
next prev parent reply other threads:[~2018-06-25 10:40 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-20 10:50 vc-dir operation is very slow on large git repositories in Emacs 26.1 Alex Harsanyi
2018-06-20 15:51 ` Eli Zaretskii
2018-06-20 22:33 ` Alex Harsanyi
2018-06-21 14:32 ` Eli Zaretskii
2018-06-22 1:12 ` Alex Harsanyi
2018-06-22 8:57 ` Eli Zaretskii
2018-06-22 9:46 ` Alex Harsanyi
2018-06-22 12:46 ` Alex Harsanyi
2018-06-22 18:24 ` Eli Zaretskii
2018-06-22 20:43 ` w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1) Stefan Monnier
2018-06-23 7:21 ` Eli Zaretskii
[not found] ` <CAH-ciFrCk=pVTDUA+8F1XZSK40Wr21yzz3u=mutJ0NM3UdojNw@mail.gmail.com>
2018-06-23 13:30 ` Fwd: " Alex Harsanyi
2018-06-23 13:32 ` Alex Harsanyi
2018-06-23 13:40 ` Eli Zaretskii
2018-06-23 14:00 ` Alex Harsanyi
2018-06-23 14:31 ` Eli Zaretskii
2018-06-30 8:46 ` Eli Zaretskii
2018-06-25 10:40 ` Alex Harsanyi [this message]
2018-06-25 11:09 ` vc-dir operation is very slow on large git repositories in Emacs 26.1 Bastian Beischer
2018-06-25 12:23 ` Alex Harsanyi
2018-06-25 15:17 ` Eli Zaretskii
2018-06-25 15:44 ` Bastian Beischer
2018-06-25 16:23 ` Eli Zaretskii
2018-06-25 22:28 ` Alex Harsanyi
2018-06-25 14:51 ` Eli Zaretskii
2018-06-22 9:04 ` Dmitry Gutov
2018-06-23 13:37 ` Eli Zaretskii
2018-06-23 18:28 ` Dmitry Gutov
2018-06-23 18:44 ` Eli Zaretskii
2018-06-25 12:55 ` Dmitry Gutov
2018-06-25 15:20 ` Eli Zaretskii
2018-06-26 15:14 ` Sergey Organov
2018-06-27 2:01 ` Dmitry Gutov
2018-06-27 2:49 ` Tom Tromey
2018-06-27 9:41 ` Dmitry Gutov
2018-06-27 15:12 ` Eli Zaretskii
2018-06-28 0:30 ` Dmitry Gutov
2018-06-28 12:55 ` Eli Zaretskii
2018-06-28 13:06 ` Dmitry Gutov
2018-06-28 13:12 ` Eli Zaretskii
2018-06-28 13:18 ` Dmitry Gutov
2018-06-28 13:38 ` Eli Zaretskii
2018-06-28 13:47 ` Dmitry Gutov
2018-06-28 14:00 ` Eli Zaretskii
2018-06-28 13:21 ` Sergey Organov
2018-06-28 13:28 ` Dmitry Gutov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAH-ciFrG2hfWwQeAPLnNP6QBkNw1wuquxgEvch=ZqfAY=hCARg@mail.gmail.com' \
--to=alexharsanyi@gmail.com \
--cc=eliz@gnu.org \
--cc=emacs-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.