From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Alex Harsanyi Newsgroups: gmane.emacs.devel Subject: Re: vc-dir operation is very slow on large git repositories in Emacs 26.1 Date: Fri, 22 Jun 2018 17:46:00 +0800 Message-ID: References: <83k1qtsbgi.fsf@gnu.org> <83zhzoqkgv.fsf@gnu.org> <83efgzqjv5.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 (1.0) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: blaine.gmane.org 1529660680 5597 195.159.176.226 (22 Jun 2018 09:44:40 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 22 Jun 2018 09:44:40 +0000 (UTC) Cc: emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Fri Jun 22 11:44:35 2018 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fWIcK-0001Kj-VM for ged-emacs-devel@m.gmane.org; Fri, 22 Jun 2018 11:44:33 +0200 Original-Received: from localhost ([::1]:60381 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWIeS-00076f-1e for ged-emacs-devel@m.gmane.org; Fri, 22 Jun 2018 05:46:44 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:35583) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWIdv-0006x2-Ko for emacs-devel@gnu.org; Fri, 22 Jun 2018 05:46:12 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fWIdu-0007Bw-L5 for emacs-devel@gnu.org; Fri, 22 Jun 2018 05:46:11 -0400 Original-Received: from mail-pg0-x22c.google.com ([2607:f8b0:400e:c05::22c]:36813) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fWIdp-00077f-5h; Fri, 22 Jun 2018 05:46:05 -0400 Original-Received: by mail-pg0-x22c.google.com with SMTP id m5-v6so2748175pgd.3; Fri, 22 Jun 2018 02:46:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=KlnCy34qCadgzg2Yls4COOQDt1qGyMXwYBC6ZdyIIos=; b=s7QEEn9QcLJLuAIOry1LBd4FJniePZVT7Qmng9266gCMAatTQW28u450Sj0BczMQ18 YfE697J3IJimRWWMy1759Te29NTxnWIaM9oY3yBlTWOMmf8zvVBx5OM5TSVZGmqgt/lD ctlAjr/eE0awPfgc7lVq5zK6+x/iDOS1lSCpY5UKw0Zl3VG6FggrD8UIR3bqMK2Inxc2 UXCdj2r8kRmtIk3a5wqbkBeIj/rRH9zbgvohrmvFw0260LvpW8WhbGf5W1NCDz0bGoLQ cx4eKP5de2Dh/SOog79lU7yGRovR1gqmqOYq5UC++eBn7qvPcEMC51qx+bIvrx8zQEJg 3SKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=KlnCy34qCadgzg2Yls4COOQDt1qGyMXwYBC6ZdyIIos=; b=D8z4jI5Ex2pd9p2OUxWLF1ram1SumsFjawKZ0cvUoxBIXhvw2HxRbfKYDzYhSj7uol YeTgCYWEGAjAB2/UwYAE22RGGfDpibnDScBMvEi5z5wwp7+iMu5kqPzw/THRaHcfF4st KQ7ScpdoyihB5+MGA5dB//oSDin0KyAZWjpQMR6lsVK6stYtvMYe/in3nf5IiFPPKhnQ DYh75D3fNnOU5R20OASIFHbFAcqkxXjpHyeljc0RvmqCv8V8J0VmB2wgfO6YjFLYeTqq VGBJmlzZ30BW3tcxe7UAxBTUAN7oLEVuS99DccZz740MuLJBGeRtnKI6G+vGXym8CxcV Ly8Q== X-Gm-Message-State: APt69E1ExwHgdrX4GhN0TF7GFvv/ysPr/U3+4hdUCpEOuWfsM5nrRTwy CuOjLrLDIqCVkSsNWGQOeRq0EtkyGVA= X-Google-Smtp-Source: ADUXVKJE29d/xEz936wHSDQaPfl1JV8iOSq8yay/0no4KBuPBUM6fsO8LCVYKh7GRmxKodct/qtieQ== X-Received: by 2002:a62:4c04:: with SMTP id z4-v6mr933471pfa.205.1529660763863; Fri, 22 Jun 2018 02:46:03 -0700 (PDT) Original-Received: from [10.25.125.47] (pa49-196-153-244.pa.vic.optusnet.com.au. [49.196.153.244]) by smtp.gmail.com with ESMTPSA id w3-v6sm13646011pfi.109.2018.06.22.02.46.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Jun 2018 02:46:03 -0700 (PDT) X-Mailer: iPhone Mail (15F79) In-Reply-To: <83efgzqjv5.fsf@gnu.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::22c X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:226587 Archived-At: On 22 Jun 2018, at 4:57 pm, Eli Zaretskii wrote: >> From: Alex Harsanyi >> Date: Fri, 22 Jun 2018 09:12:29 +0800 >> Cc: emacs-devel@gnu.org >>=20 >> I set `vc-command-messages`, which prints out the subprocesses run by >> `vc-do-command` and also updated `vc-do-command` to show the time it took= for >> each process. It turns out that Emacs 26.1 does run one extra command: >>=20 >> In Emacs 25.2, a "vc-dir" command runs the following: >>=20 >> Running git --no-pager update-index --refresh . in background... done >> 0.48 seconds >> Running git --no-pager diff-index --relative -z -M HEAD -- . in >> background... done 0.42 seconds >> Running git --no-pager ls-files -z -o --directory --no-empty-directory >> --exclude-standard -- . in background... done 0.10 seconds >>=20 >> In Emacs 26.2, a "vc-dir" command runs the following: >>=20 >> Running git --no-pager update-index --refresh . in background... done >> 0.46 seconds >> Running git --no-pager diff-index --relative -z -M HEAD -- . in >> background... done 0.46 seconds >> Running git --no-pager ls-files -z -c -s -- . in background... done 4.49 s= econds >> Running git --no-pager ls-files -z -o --directory --no-empty-directory >> --exclude-standard -- . in background... done 0.09 seconds >>=20 >> The extra command is "git --no-pager ls-files -z -c -s -- ." and it takes= 4.5 >> seconds to run (I ran each test multiple times and the times are consiste= nt). >> However, when I run that command in a terminal, it finishes in less than a= >> second. >=20 > But the times you cited originally were much longer: 2 min. What > caused such a stark difference, from 2 min to just 4.5 sec? The repository at my work place has more than 50000 files and vc-dir takes 2= minutes (measured using a stopwatch) for that repo.=20 These tests I did at home against the Emacs repository which had approximate= ly 3000 files. Originally I said this takes about 7 seconds, but that was =E2= =80=9Cmeasured=E2=80=9D by counting slowly in my head, these latest numbers a= ctually measure the exact time.=20 >=20 >> An Emacs-Lisp program to run this script, but ignore all output, it takes= 16 >> seconds, in both 25.2 and 26.1: >=20 > Once again, 16 sec is much shorter than 2 min you originally cited. The ls-files command from for my work repo generates 6.5 Mb of data, my test= program only generated 1 Mb of data. >=20 >> It seems reading output from a process in Emacs is somehow limited to 64 K= b / >> second -- I verified this with procmon. It looks like the behavior is th= e >> same in 25.2 and 26.1, it is just that in vc-dir 25.2 does not run a git >> command that produces a lot of output. >=20 > Does it help to set w32-pipe-buffer-size to some large value, like > 16384, and/or decrease w32-pipe-read-delay? I will try that tonight. Thanks for the suggestion.=20 Alex >=20 > Thanks.