From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Alex Harsanyi Newsgroups: gmane.emacs.devel Subject: Re: vc-dir operation is very slow on large git repositories in Emacs 26.1 Date: Fri, 22 Jun 2018 09:12:29 +0800 Message-ID: References: <83k1qtsbgi.fsf@gnu.org> <83zhzoqkgv.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Trace: blaine.gmane.org 1529629864 14546 195.159.176.226 (22 Jun 2018 01:11:04 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 22 Jun 2018 01:11:04 +0000 (UTC) Cc: emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Fri Jun 22 03:11:00 2018 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fWAbL-0003fd-US for ged-emacs-devel@m.gmane.org; Fri, 22 Jun 2018 03:11:00 +0200 Original-Received: from localhost ([::1]:58266 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWAdS-00064i-VZ for ged-emacs-devel@m.gmane.org; Thu, 21 Jun 2018 21:13:11 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:39452) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWAcs-00064d-8I for emacs-devel@gnu.org; Thu, 21 Jun 2018 21:12:35 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fWAcr-0006QO-4U for emacs-devel@gnu.org; Thu, 21 Jun 2018 21:12:34 -0400 Original-Received: from mail-it0-x230.google.com ([2607:f8b0:4001:c0b::230]:40204) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fWAcp-0006Oy-Ey; Thu, 21 Jun 2018 21:12:31 -0400 Original-Received: by mail-it0-x230.google.com with SMTP id 188-v6so670864ita.5; Thu, 21 Jun 2018 18:12:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=K3gOl2OhJO99gVEvisdyezY7AJNKww/T7jW8ctve6QA=; b=cmys0qn42TAY+5N4UCSBySrb3mlAH+tX5mNvVgrPaoGjrKSTv7IU1Hwc99HL++zu2q 6PxBY/RR7h3/R9TxzM8YiEt73D+6dusHoaz5/beXpoOX+cBdP6TfUtr1fFMRGRXSSTh7 wylE7XwsaJU4GPR5jJzZH9N7cPE6iTIjRz6sEUTibtU3580AT2Fuh37R5poT2/P5b3kK 4CBos8WXRxp15SgTQ2fyrfO9T16Snudh1z+8OkzRLrHhxECbwfy4WE0mrwxX+tXRfbkU 5qalpIU19C+Su3zOKkIk+CA50ZCGAV9khRVgF6bWHCRLz2Yu5BFhTt75R32e4vaYp+na kk9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=K3gOl2OhJO99gVEvisdyezY7AJNKww/T7jW8ctve6QA=; b=QBVn7gksmZQw/M3sbUyTOxJQhLJX2Q3GqFjNDQY2a0skwNy5cTy8ehrd5QlcI576pT EFncDhH0yODBEo0sIGVCGLpJ8eLon/L0gcikZhXouLftz+Is7dgpwBdV82+LZXcXQrMk R0wAemkzta7ceJx7vjnZaw4zr/1ICtT3B2XYWD87CoG+aree6qGmUVzPzWh6FZdX0t3E le5QxfFJndRtGkCuC1czCxO7VInXLWQZAObPE7jGus2s7+4LSwBOogEryIkXWpeLEhuo DdEKLV+7MOtv3Vnq0UvRvGcAaxpOOcFQi5oUH22I1V9/JSbfg+8X5ara/1due6JWogL4 SLPg== X-Gm-Message-State: APt69E0/5t2CMY3Htqc0oOLlrd0/NdlQa4LzI8HB45nnSmrTeGpgtRNM ZEra5g+C101QjuYpKNJtlbRs2rtl0vkTxYpzZKUFFSqK X-Google-Smtp-Source: ADUXVKKzXoUl8l1IP0GPybyNN2i8uqHV2R2MgEBb/ympvWiggPvmL+LRgItmxOlkrcFP4BleNAtiBvx1T5Yo8rCy8S8= X-Received: by 2002:a24:5a11:: with SMTP id v17-v6mr7422334ita.40.1529629950049; Thu, 21 Jun 2018 18:12:30 -0700 (PDT) Original-Received: by 2002:a6b:4514:0:0:0:0:0 with HTTP; Thu, 21 Jun 2018 18:12:29 -0700 (PDT) In-Reply-To: <83zhzoqkgv.fsf@gnu.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4001:c0b::230 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:226579 Archived-At: On Thu, Jun 21, 2018 at 10:32 PM, Eli Zaretskii wrote: > > Can you step with Edebug through the vc-git.el function where this > command is invoked, and see if it gets invoked in Emacs 25? I set `vc-command-messages`, which prints out the subprocesses run by `vc-do-command` and also updated `vc-do-command` to show the time it took for each process. It turns out that Emacs 26.1 does run one extra command: In Emacs 25.2, a "vc-dir" command runs the following: Running git --no-pager update-index --refresh . in background... done 0.48 seconds Running git --no-pager diff-index --relative -z -M HEAD -- . in background... done 0.42 seconds Running git --no-pager ls-files -z -o --directory --no-empty-directory --exclude-standard -- . in background... done 0.10 seconds In Emacs 26.2, a "vc-dir" command runs the following: Running git --no-pager update-index --refresh . in background... done 0.46 seconds Running git --no-pager diff-index --relative -z -M HEAD -- . in background... done 0.46 seconds Running git --no-pager ls-files -z -c -s -- . in background... done 4.49 seconds Running git --no-pager ls-files -z -o --directory --no-empty-directory --exclude-standard -- . in background... done 0.09 seconds The extra command is "git --no-pager ls-files -z -c -s -- ." and it takes 4.5 seconds to run (I ran each test multiple times and the times are consistent). However, when I run that command in a terminal, it finishes in less than a second. The only thing special about that command is that it produce a lot of output, so I wrote the following test: A script to generate lots of output (written in Perl): $line = "a" x 1024; $nlines = 1024; while ($nlines-- > 0) { print $line; } Running the perl script in the terminal takes 0.4 seconds if I let it print all output to the terminal and takes 0.047 seconds if I redirect it to a file: $ time perl gen-output.pl [... lots of output omitted ...] real 0m0.388s user 0m0.000s sys 0m0.030s $ time perl gen-output.pl > x real 0m0.047s user 0m0.015s sys 0m0.015s An Emacs-Lisp program to run this script, but ignore all output, it takes 16 seconds, in both 25.2 and 26.1: ;;; -*- lexical-binding: t -*- (defun time-delta (start end) (destructuring-bind (_shigh slow susec spsec) start (destructuring-bind (_ehigh elow eusec epsec) end (let ((s (+ (* 1.0 slow) (* 1e-6 susec) (* 1e-12 spsec))) (e (+ (* 1.0 elow) (* 1e-6 eusec) (* 1e-12 epsec)))) (- e s))))) (defun time-process () (interactive) (let* ((start (current-time)) (buf (get-buffer-create " *AH-TMP*")) (proc (start-file-process "AH-TMP" buf "perl" "gen-output.pl"))) (set-process-sentinel proc (lambda (p _code) (when (eq (process-status p) 'exit) (let ((delta (time-delta start (current-time)))) (message "time-process: it took %.2f seconds" delta))))) (set-process-filter proc #'ignore))) It seems reading output from a process in Emacs is somehow limited to 64 Kb / second -- I verified this with procmon. It looks like the behavior is the same in 25.2 and 26.1, it is just that in vc-dir 25.2 does not run a git command that produces a lot of output. > >> Both versions were downloaded as compiled binaries from the >> gnu.org website and not modified in any way -- this happens when >> I start Emacs with "-Q -no-site-file". > > Are they both 64-bit builds (or both 32-bit)? Both are 64 bit builds, `emacs-version` shows: GNU Emacs 26.1 (build 1, x86_64-w64-mingw32) of 2018-05-30 GNU Emacs 25.2.1 (x86_64-w64-mingw32) of 2017-04-25