From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Bastian Beischer Newsgroups: gmane.emacs.devel Subject: Re: vc-dir operation is very slow on large git repositories in Emacs 26.1 Date: Mon, 25 Jun 2018 17:44:21 +0200 Message-ID: References: <83k1qtsbgi.fsf@gnu.org> <83zhzoqkgv.fsf@gnu.org> <83efgzqjv5.fsf@gnu.org> <83wouqptm6.fsf@gnu.org> <83lgb2sxnr.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Trace: blaine.gmane.org 1529941350 19936 195.159.176.226 (25 Jun 2018 15:42:30 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 25 Jun 2018 15:42:30 +0000 (UTC) Cc: Alex Harsanyi , Emacs-Devel To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Jun 25 17:42:25 2018 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fXTdJ-00054f-JE for ged-emacs-devel@m.gmane.org; Mon, 25 Jun 2018 17:42:25 +0200 Original-Received: from localhost ([::1]:47883 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXTfQ-0005on-NF for ged-emacs-devel@m.gmane.org; Mon, 25 Jun 2018 11:44:36 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:48580) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXTfG-0005ms-5I for emacs-devel@gnu.org; Mon, 25 Jun 2018 11:44:27 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXTfF-00069T-1N for emacs-devel@gnu.org; Mon, 25 Jun 2018 11:44:26 -0400 Original-Received: from mail-ua0-x244.google.com ([2607:f8b0:400c:c08::244]:45028) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fXTfD-00068o-Go; Mon, 25 Jun 2018 11:44:23 -0400 Original-Received: by mail-ua0-x244.google.com with SMTP id v15-v6so2051910ual.11; Mon, 25 Jun 2018 08:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=APx5oLgzpbnQMPtliHiiv431N8ZRbDghgLfylCNwUd8=; b=K5XGROBQVkflLIvmsLunUpDeloHlAxoVge7L1iKU7SpQ6RdG2LyMsLNvmeHmvRBT89 x60LFWGLuBpyRIIqEMy430xdwT2LPoq2fsyUPkdb8RDhqXDlb6dubOQfK6gGUnkXm/N9 POlADtMeYmSLLdmaQvc/XpE25jzrozI4sxpbbDWda/7Q+r6SnQzwt39EJWo1VStcRhp4 lrvuBZDPL/fFjHpA3pa/EWDphlEhAE/AdiCIvM0doKGSWJX0k0CvIsrBUnU/FaH2oIp7 8h5T6IuMGVXBAsNfFzgIcAKvaV/yugQ4v6X3UP7itQ+ZGOSsoIoptPQwp5vt8djStN2S Irng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=APx5oLgzpbnQMPtliHiiv431N8ZRbDghgLfylCNwUd8=; b=bumdPaVjRO9z0JOwOBOJChl29jqhvbe9xWJDlgirID/iQ+m+zYP4i8CaT+nngflKbv FEBDJ3F8hqOiFyYGOzgj+R2b5B2qYkxRvE6uIb1W3qpySTe5SIjPw1sXS2pzhfYeEzHE uiFGU3iHN+AccqGv46A7wB3fvhOIyIq7abxHNjC+PzipVVR3EJ/Ne61GXNdiD3m+XX8n qLUB+B94aCxJKBJj6lDhB9/5e0qUC/G72EaAJrNQVmCeobgjVVvv9q9ZLXd8HsyXrLcN P0kPRNtyzEAVrUxQ0PUZvDqXcWvtAQzJw+htoFzKCl6b2twptMkvkX/GJeFa31Jl537K TipA== X-Gm-Message-State: APt69E1EKJfCjcJBLrVREGRe/M0ZWhZT5sAdYakPnf8kd/Hvw9zb5583 MxQNTf2qt6beRUmSP3lH2DxI/iiHbz7b9GSuCdhioQ== X-Google-Smtp-Source: ADUXVKLSDTwRfcuuWGr63lFP5vMtcTlKV9/yTzqjK5R/kYaYNgRIKqN1RuN6kNOLvtgvImik5bU9vbctfDS1zcmrEcg= X-Received: by 2002:ab0:1047:: with SMTP id g7-v6mr720585uab.105.1529941462171; Mon, 25 Jun 2018 08:44:22 -0700 (PDT) Original-Received: by 2002:ab0:5e08:0:0:0:0:0 with HTTP; Mon, 25 Jun 2018 08:44:21 -0700 (PDT) In-Reply-To: <83lgb2sxnr.fsf@gnu.org> X-Google-Sender-Auth: w3SsHguRzVNOCihskRT-yOsOrpk X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400c:c08::244 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:226711 Archived-At: On Mon, Jun 25, 2018 at 5:17 PM, Eli Zaretskii wrote: >> From: Bastian Beischer >> Date: Mon, 25 Jun 2018 13:09:37 +0200 >> Cc: Eli Zaretskii , Emacs-Devel >> >> >From these timings it appears that you have been using a 4096 byte >> buffer. > > Yes, the default buffer size of a pipe on Windows is 4KB. > >> Assume that all the delay in the 83.7 seconds is due to 50ms waits, >> one such wait whenever the buffer is full. This is based on the fact >> that you observe a ~proportional slowdown when increasing the >> read-delay. Then this means that the buffer was full 1674 times, which >> means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096 >> bytes. So the predicted approximate delay for a message of size S is >> >> D = (S / B) * W >> >> where B is the buffer size (4096) and W is the w32-pipe-read-delay >> (0.05 s). Based on that logic it appears that setting >> w32-pipe-buffer-size to 16384 should give you a ~four times smaller >> total delay - especially when w32-pipe-read-delay is 50 ms, for >> example. Unless the OS refuses to provide you with such a buffer size? > > I think you ignore a few factors here. Yes indeed, but the fact that one can go to ~1s by reducing the delay suggests that all of this can be ignored to first order. It looks like the input rate to the pipe can be considered infinite in a good approximation. > Are you familiar with how > Emacs on Windows reads data from async subprocesses? If not, I > suggest to read the large comment around line 800 of w32proc.c and > study the code it describes. No I'm not familiar with the actual code - you are very right to point me to it and I will study it. I'm just making some very naive observations :) > > With 50-msec delay, it is reasonable to assume that the pipe's buffer > gets filled while the reader thread waits for 50 msec, so most of the > time is indeed due to these waits. But with zero waits, its quite > possible that Emacs reads the stuff from the pipe as fast as Git > writes to its other end, because the reading can begin very soon after > the first byte is written to the pipe (in reality, writes are > probably buffered, so the first chunk is much larger than 1 byte). If > Emacs empties the pipe as fast as Git writes to it or faster, the size > of the pipe will have (almost) no effect whatsoever. I agree with all of this, but the point is that _if_ with 50 msec delay the buffer get's filled completely, we should assume that a buffer twice as large woud also get filled completely, provided that the input rate is near infinite. I think that assumption is valid because the total time is linear with the delay down to delays of order 5 msec at least. That should result in a data rate which is twice as large. Therefore I don't understand how the timing can be independent of the buffer size. But maybe studying the code will make me understand the situation :) Looking at commit 58a622d473112f8ff5b4bdb3e49bc6573dfd3404 (where you introduced w32-pipe-buffer-size) I can see a code path that results in resetting the buffer size to 4 KB. Maybe that get's taken? > >> Unless the OS refuses to provide you with such a buffer size? > > No, there should be no such problem. > > Enlarging the pipe buffer size is (IME) only needed when Emacs needs > to _write_ large amounts of data to the subprocesses, and the > subprocesses doesn't keep up. We write to the pipe in the main > thread, so with small enough buffer and slow enough reading by the > subprocesses will eventually cause Emacs to hang, which is not really > user-friendly. See bug#22344 and bug#24143, for example.