From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: vc-dir operation is very slow on large git repositories in Emacs 26.1 Date: Mon, 25 Jun 2018 18:17:44 +0300 Message-ID: <83lgb2sxnr.fsf@gnu.org> References: <83k1qtsbgi.fsf@gnu.org> <83zhzoqkgv.fsf@gnu.org> <83efgzqjv5.fsf@gnu.org> <83wouqptm6.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org X-Trace: blaine.gmane.org 1529939757 6543 195.159.176.226 (25 Jun 2018 15:15:57 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 25 Jun 2018 15:15:57 +0000 (UTC) Cc: alexharsanyi@gmail.com, emacs-devel@gnu.org To: Bastian Beischer Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Jun 25 17:15:53 2018 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fXTDc-0001aW-RP for ged-emacs-devel@m.gmane.org; Mon, 25 Jun 2018 17:15:52 +0200 Original-Received: from localhost ([::1]:47717 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXTFk-0005EC-0V for ged-emacs-devel@m.gmane.org; Mon, 25 Jun 2018 11:18:04 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:40256) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXTFV-0005Cz-46 for emacs-devel@gnu.org; Mon, 25 Jun 2018 11:17:50 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXTFR-0004j9-TR for emacs-devel@gnu.org; Mon, 25 Jun 2018 11:17:49 -0400 Original-Received: from fencepost.gnu.org ([2001:4830:134:3::e]:41717) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXTFR-0004j3-PS; Mon, 25 Jun 2018 11:17:45 -0400 Original-Received: from [176.228.60.248] (port=3386 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1fXTFR-00043W-76; Mon, 25 Jun 2018 11:17:45 -0400 In-reply-to: (message from Bastian Beischer on Mon, 25 Jun 2018 13:09:37 +0200) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:226708 Archived-At: > From: Bastian Beischer > Date: Mon, 25 Jun 2018 13:09:37 +0200 > Cc: Eli Zaretskii , Emacs-Devel > > >From these timings it appears that you have been using a 4096 byte > buffer. Yes, the default buffer size of a pipe on Windows is 4KB. > Assume that all the delay in the 83.7 seconds is due to 50ms waits, > one such wait whenever the buffer is full. This is based on the fact > that you observe a ~proportional slowdown when increasing the > read-delay. Then this means that the buffer was full 1674 times, which > means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096 > bytes. So the predicted approximate delay for a message of size S is > > D = (S / B) * W > > where B is the buffer size (4096) and W is the w32-pipe-read-delay > (0.05 s). Based on that logic it appears that setting > w32-pipe-buffer-size to 16384 should give you a ~four times smaller > total delay - especially when w32-pipe-read-delay is 50 ms, for > example. Unless the OS refuses to provide you with such a buffer size? I think you ignore a few factors here. Are you familiar with how Emacs on Windows reads data from async subprocesses? If not, I suggest to read the large comment around line 800 of w32proc.c and study the code it describes. With 50-msec delay, it is reasonable to assume that the pipe's buffer gets filled while the reader thread waits for 50 msec, so most of the time is indeed due to these waits. But with zero waits, its quite possible that Emacs reads the stuff from the pipe as fast as Git writes to its other end, because the reading can begin very soon after the first byte is written to the pipe (in reality, writes are probably buffered, so the first chunk is much larger than 1 byte). If Emacs empties the pipe as fast as Git writes to it or faster, the size of the pipe will have (almost) no effect whatsoever. > Unless the OS refuses to provide you with such a buffer size? No, there should be no such problem. Enlarging the pipe buffer size is (IME) only needed when Emacs needs to _write_ large amounts of data to the subprocesses, and the subprocesses doesn't keep up. We write to the pipe in the main thread, so with small enough buffer and slow enough reading by the subprocesses will eventually cause Emacs to hang, which is not really user-friendly. See bug#22344 and bug#24143, for example.