Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii <eliz@gnu.org>]

> From: Bastian Beischer <bastian.beischer@rwth-aachen.de>
> Date: Mon, 25 Jun 2018 13:09:37 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-Devel <emacs-devel@gnu.org>
> 
> >From these timings it appears that you have been using a 4096 byte
> buffer.

Yes, the default buffer size of a pipe on Windows is 4KB.

> Assume that all the delay in the 83.7 seconds is due to 50ms waits,
> one such wait whenever the buffer is full. This is based on the fact
> that you observe a ~proportional slowdown when increasing the
> read-delay. Then this means that the buffer was full 1674 times, which
> means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096
> bytes. So the predicted approximate delay for a message of size S is
> 
> D = (S / B) * W
> 
> where B is the buffer size (4096) and W is the w32-pipe-read-delay
> (0.05 s). Based on that logic it appears that setting
> w32-pipe-buffer-size to 16384 should give you a ~four times smaller
> total delay - especially when w32-pipe-read-delay is 50 ms, for
> example. Unless the OS refuses to provide you with such a buffer size?

I think you ignore a few factors here.  Are you familiar with how
Emacs on Windows reads data from async subprocesses?  If not, I
suggest to read the large comment around line 800 of w32proc.c and
study the code it describes.

With 50-msec delay, it is reasonable to assume that the pipe's buffer
gets filled while the reader thread waits for 50 msec, so most of the
time is indeed due to these waits.  But with zero waits, its quite
possible that Emacs reads the stuff from the pipe as fast as Git
writes to its other end, because the reading can begin very soon after
the first byte is written to the pipe (in reality, writes are
probably buffered, so the first chunk is much larger than 1 byte).  If
Emacs empties the pipe as fast as Git writes to it or faster, the size
of the pipe will have (almost) no effect whatsoever.

> Unless the OS refuses to provide you with such a buffer size?

No, there should be no such problem.

Enlarging the pipe buffer size is (IME) only needed when Emacs needs
to _write_ large amounts of data to the subprocesses, and the
subprocesses doesn't keep up.  We write to the pipe in the main
thread, so with small enough buffer and slow enough reading by the
subprocesses will eventually cause Emacs to hang, which is not really
user-friendly.  See bug#22344 and bug#24143, for example.