Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Bastian Beischer <bastian.beischer@rwth-aachen.de>]

On Mon, Jun 25, 2018 at 5:17 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Bastian Beischer <bastian.beischer@rwth-aachen.de>
>> Date: Mon, 25 Jun 2018 13:09:37 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-Devel <emacs-devel@gnu.org>
>>
>> >From these timings it appears that you have been using a 4096 byte
>> buffer.
>
> Yes, the default buffer size of a pipe on Windows is 4KB.
>
>> Assume that all the delay in the 83.7 seconds is due to 50ms waits,
>> one such wait whenever the buffer is full. This is based on the fact
>> that you observe a ~proportional slowdown when increasing the
>> read-delay. Then this means that the buffer was full 1674 times, which
>> means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096
>> bytes. So the predicted approximate delay for a message of size S is
>>
>> D = (S / B) * W
>>
>> where B is the buffer size (4096) and W is the w32-pipe-read-delay
>> (0.05 s). Based on that logic it appears that setting
>> w32-pipe-buffer-size to 16384 should give you a ~four times smaller
>> total delay - especially when w32-pipe-read-delay is 50 ms, for
>> example. Unless the OS refuses to provide you with such a buffer size?
>
> I think you ignore a few factors here.

Yes indeed, but the fact that one can go to ~1s by reducing the delay
suggests that all of this can be ignored to first order. It looks like
the input rate to the pipe can be considered infinite in a good
approximation.

> Are you familiar with how
> Emacs on Windows reads data from async subprocesses?  If not, I
> suggest to read the large comment around line 800 of w32proc.c and
> study the code it describes.

No I'm not familiar with the actual code - you are very right to point
me to it and I will study it. I'm just making some very naive
observations :)

>
> With 50-msec delay, it is reasonable to assume that the pipe's buffer
> gets filled while the reader thread waits for 50 msec, so most of the
> time is indeed due to these waits.  But with zero waits, its quite
> possible that Emacs reads the stuff from the pipe as fast as Git
> writes to its other end, because the reading can begin very soon after
> the first byte is written to the pipe (in reality, writes are
> probably buffered, so the first chunk is much larger than 1 byte).  If
> Emacs empties the pipe as fast as Git writes to it or faster, the size
> of the pipe will have (almost) no effect whatsoever.

I agree with all of this, but the point is that _if_ with 50 msec
delay the buffer get's filled completely, we should assume that a
buffer twice as large woud also get filled completely, provided that
the input rate is near infinite. I think that assumption is valid
because the total time is linear with the delay down to delays of
order 5 msec at least. That should result in a data rate which is
twice as large. Therefore I don't understand how the timing can be
independent of the buffer size. But maybe studying the code will make
me understand the situation :)

Looking at commit 58a622d473112f8ff5b4bdb3e49bc6573dfd3404 (where you
introduced w32-pipe-buffer-size) I can see a code path that results in
resetting the buffer size to 4 KB. Maybe that get's taken?

>
>> Unless the OS refuses to provide you with such a buffer size?
>
> No, there should be no such problem.
>
> Enlarging the pipe buffer size is (IME) only needed when Emacs needs
> to _write_ large amounts of data to the subprocesses, and the
> subprocesses doesn't keep up.  We write to the pipe in the main
> thread, so with small enough buffer and slow enough reading by the
> subprocesses will eventually cause Emacs to hang, which is not really
> user-friendly.  See bug#22344 and bug#24143, for example.