vc-dir operation is very slow on large git repositories in Emacs 26.1

vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

I have a somewhat large git repository at work (51556 files) and
Emacs 26.1 is very slow when updating the *vc-dir* buffer: I
start emacs with the "-Q -no-site-file" and it takes
approximately 2 minutes to update the vc-dir buffer. Emacs 25.2,
on the same machine and repository, will complete this operation
in a few seconds.  This is on a Windows machine and I use the
prebuilt binaries downloaded from the gnu.org website for both
versions.

The operation that takes a long time is the ls-files
operation, "git --no-pager ls-files -z -c -s --", but running
that command in a terminal completes in less than a second.

The rest of Emacs is fully responsive and I can continue editing
while the update is running in the *vc-dir* buffer.  Also other
version control operations (diff, check-in) are responsive.

I cloned the Emacs git repository (which has 3683 files) and a
vc-dir operation in Emacs 26.1 takes 7 seconds on my machine,
compared to less than 1 second in Emacs 25.2.  I noticed this
speed difference on my home computer as well.

I was wondering if others have noticed this issue?

If not, I would appreciate some suggestions on how to further
diagnose the cause of this issue, as it seems it is related to my
computer.

Thanks,
Alex.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Date: Wed, 20 Jun 2018 18:50:12 +0800
> 
> I have a somewhat large git repository at work (51556 files) and
> Emacs 26.1 is very slow when updating the *vc-dir* buffer: I
> start emacs with the "-Q -no-site-file" and it takes
> approximately 2 minutes to update the vc-dir buffer. Emacs 25.2,
> on the same machine and repository, will complete this operation
> in a few seconds.  This is on a Windows machine and I use the
> prebuilt binaries downloaded from the gnu.org website for both
> versions.
> 
> The operation that takes a long time is the ls-files
> operation, "git --no-pager ls-files -z -c -s --", but running
> that command in a terminal completes in less than a second.

So you are saying that in Emacs 25.2 this slow Git command was not
invoked in your scenario?

> I cloned the Emacs git repository (which has 3683 files) and a
> vc-dir operation in Emacs 26.1 takes 7 seconds on my machine,
> compared to less than 1 second in Emacs 25.2.  I noticed this
> speed difference on my home computer as well.

FWIW, I get about 2 sec for this in the Emacs repository, and I see no
perceptible difference between Emacs 25.2 and Emacs 26.1.

> If not, I would appreciate some suggestions on how to further
> diagnose the cause of this issue, as it seems it is related to my
> computer.

Where did you get the Emacs binaries for the 2 above-mentioned
versions?

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On Wed, Jun 20, 2018 at 11:51 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>> Date: Wed, 20 Jun 2018 18:50:12 +0800
>>
>> I have a somewhat large git repository at work (51556 files) and
>> Emacs 26.1 is very slow when updating the *vc-dir* buffer: I
>> start emacs with the "-Q -no-site-file" and it takes
>> approximately 2 minutes to update the vc-dir buffer. Emacs 25.2,
>> on the same machine and repository, will complete this operation
>> in a few seconds.  This is on a Windows machine and I use the
>> prebuilt binaries downloaded from the gnu.org website for both
>> versions.
>>
>> The operation that takes a long time is the ls-files
>> operation, "git --no-pager ls-files -z -c -s --", but running
>> that command in a terminal completes in less than a second.
>
> So you are saying that in Emacs 25.2 this slow Git command was not
> invoked in your scenario?
>

I do not say that.  Emacs 25.2 completes too fast to be able to
figure what is going on.  I suspect 25.2 uses the same git
command though, as the command itself completes quickly.

>> I cloned the Emacs git repository (which has 3683 files) and a
>> vc-dir operation in Emacs 26.1 takes 7 seconds on my machine,
>> compared to less than 1 second in Emacs 25.2.  I noticed this
>> speed difference on my home computer as well.
>
> FWIW, I get about 2 sec for this in the Emacs repository, and I see no
> perceptible difference between Emacs 25.2 and Emacs 26.1.
>

I suspect this might be a problem with my environment, but I do
not know what.  It also happens on two machines: my work one and
my home one.  I would appreciate any hints on how to diagnose
this.

>> If not, I would appreciate some suggestions on how to further
>> diagnose the cause of this issue, as it seems it is related to my
>> computer.
>
> Where did you get the Emacs binaries for the 2 above-mentioned
> versions?

Both versions were downloaded as compiled binaries from the
gnu.org website and not modified in any way -- this happens when
I start Emacs with "-Q -no-site-file".

Alex.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Date: Thu, 21 Jun 2018 06:33:57 +0800
> Cc: emacs-devel@gnu.org
> 
> >> The operation that takes a long time is the ls-files
> >> operation, "git --no-pager ls-files -z -c -s --", but running
> >> that command in a terminal completes in less than a second.
> >
> > So you are saying that in Emacs 25.2 this slow Git command was not
> > invoked in your scenario?
> 
> I do not say that.  Emacs 25.2 completes too fast to be able to
> figure what is going on.  I suspect 25.2 uses the same git
> command though, as the command itself completes quickly.

Can you step with Edebug through the vc-git.el function where this
command is invoked, and see if it gets invoked in Emacs 25?

> I suspect this might be a problem with my environment, but I do
> not know what.  It also happens on two machines: my work one and
> my home one.  I would appreciate any hints on how to diagnose
> this.

One possibility is to run "M-x profile-start" before invoking vc-dir,
then run "M-x profile-report" after it finishes, and look at the
profile.  If vc-dir takes 2 minutes, the profiler should be able to
collect enough evidence to point at the guilty parties.  For best
results, load vc-git.el manually, so that Emacs runs the interpreted
Lisp code, not byte-compiled code.

> Both versions were downloaded as compiled binaries from the
> gnu.org website and not modified in any way -- this happens when
> I start Emacs with "-Q -no-site-file".

Are they both 64-bit builds (or both 32-bit)?

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On Thu, Jun 21, 2018 at 10:32 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> Can you step with Edebug through the vc-git.el function where this
> command is invoked, and see if it gets invoked in Emacs 25?

I set `vc-command-messages`, which prints out the subprocesses run by
`vc-do-command` and also updated `vc-do-command` to show the time it took for
each process.  It turns out that Emacs 26.1 does run one extra command:

In Emacs 25.2, a "vc-dir" command runs the following:

Running git --no-pager update-index --refresh . in background... done
0.48 seconds
Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.42 seconds
Running git --no-pager ls-files -z -o --directory --no-empty-directory
--exclude-standard -- . in background... done 0.10 seconds

In Emacs 26.2, a "vc-dir" command runs the following:

Running git --no-pager update-index --refresh . in background... done
0.46 seconds
Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.46 seconds
Running git --no-pager ls-files -z -c -s -- . in background... done 4.49 seconds
Running git --no-pager ls-files -z -o --directory --no-empty-directory
--exclude-standard -- . in background... done 0.09 seconds

The extra command is "git --no-pager ls-files -z -c -s -- ." and it takes 4.5
seconds to run (I ran each test multiple times and the times are consistent).
However, when I run that command in a terminal, it finishes in less than a
second.

The only thing special about that command is that it produce a lot of output,
so I wrote the following test:

A script to generate lots of output (written in Perl):

    $line = "a" x 1024;
    $nlines = 1024;
    while ($nlines-- > 0) {
        print $line;
    }

Running the perl script in the terminal takes 0.4 seconds if I let it print
all output to the terminal and takes 0.047 seconds if I redirect it to a file:

    $ time perl gen-output.pl
    [... lots of output omitted ...]
    real    0m0.388s
    user    0m0.000s
    sys     0m0.030s

    $ time perl gen-output.pl > x

    real    0m0.047s
    user    0m0.015s
    sys     0m0.015s

An Emacs-Lisp program to run this script, but ignore all output, it takes 16
seconds, in both 25.2 and 26.1:

    ;;;  -*- lexical-binding: t -*-

    (defun time-delta (start end)
      (destructuring-bind (_shigh slow susec spsec) start
        (destructuring-bind (_ehigh elow eusec epsec) end
          (let ((s (+ (* 1.0 slow) (* 1e-6 susec) (* 1e-12 spsec)))
                (e (+ (* 1.0 elow) (* 1e-6 eusec) (* 1e-12 epsec))))
            (- e s)))))

    (defun time-process ()
      (interactive)
      (let* ((start (current-time))
             (buf (get-buffer-create " *AH-TMP*"))
             (proc (start-file-process "AH-TMP" buf "perl" "gen-output.pl")))
        (set-process-sentinel
         proc
         (lambda (p _code)
           (when (eq (process-status p) 'exit)
             (let ((delta (time-delta start (current-time))))
               (message "time-process: it took %.2f seconds" delta)))))
        (set-process-filter proc #'ignore)))

It seems reading output from a process in Emacs is somehow limited to 64 Kb /
second -- I verified this with procmon.  It looks like the behavior is the
same in 25.2 and 26.1, it is just that in vc-dir 25.2 does not run a git
command that produces a lot of output.

>
>> Both versions were downloaded as compiled binaries from the
>> gnu.org website and not modified in any way -- this happens when
>> I start Emacs with "-Q -no-site-file".
>
> Are they both 64-bit builds (or both 32-bit)?

Both are 64 bit builds, `emacs-version` shows:

GNU Emacs 26.1 (build 1, x86_64-w64-mingw32) of 2018-05-30
GNU Emacs 25.2.1 (x86_64-w64-mingw32) of 2017-04-25

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Date: Fri, 22 Jun 2018 09:12:29 +0800
> Cc: emacs-devel@gnu.org
> 
> I set `vc-command-messages`, which prints out the subprocesses run by
> `vc-do-command` and also updated `vc-do-command` to show the time it took for
> each process.  It turns out that Emacs 26.1 does run one extra command:
> 
> In Emacs 25.2, a "vc-dir" command runs the following:
> 
> Running git --no-pager update-index --refresh . in background... done
> 0.48 seconds
> Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.42 seconds
> Running git --no-pager ls-files -z -o --directory --no-empty-directory
> --exclude-standard -- . in background... done 0.10 seconds
> 
> In Emacs 26.2, a "vc-dir" command runs the following:
> 
> Running git --no-pager update-index --refresh . in background... done
> 0.46 seconds
> Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.46 seconds
> Running git --no-pager ls-files -z -c -s -- . in background... done 4.49 seconds
> Running git --no-pager ls-files -z -o --directory --no-empty-directory
> --exclude-standard -- . in background... done 0.09 seconds
> 
> The extra command is "git --no-pager ls-files -z -c -s -- ." and it takes 4.5
> seconds to run (I ran each test multiple times and the times are consistent).
> However, when I run that command in a terminal, it finishes in less than a
> second.

But the times you cited originally were much longer: 2 min.  What
caused such a stark difference, from 2 min to just 4.5 sec?

> An Emacs-Lisp program to run this script, but ignore all output, it takes 16
> seconds, in both 25.2 and 26.1:

Once again, 16 sec is much shorter than 2 min you originally cited.

> It seems reading output from a process in Emacs is somehow limited to 64 Kb /
> second -- I verified this with procmon.  It looks like the behavior is the
> same in 25.2 and 26.1, it is just that in vc-dir 25.2 does not run a git
> command that produces a lot of output.

Does it help to set w32-pipe-buffer-size to some large value, like
16384, and/or decrease w32-pipe-read-delay?

Thanks.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On 22 Jun 2018, at 4:57 pm, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>> Date: Fri, 22 Jun 2018 09:12:29 +0800
>> Cc: emacs-devel@gnu.org
>> 
>> I set `vc-command-messages`, which prints out the subprocesses run by
>> `vc-do-command` and also updated `vc-do-command` to show the time it took for
>> each process.  It turns out that Emacs 26.1 does run one extra command:
>> 
>> In Emacs 25.2, a "vc-dir" command runs the following:
>> 
>> Running git --no-pager update-index --refresh . in background... done
>> 0.48 seconds
>> Running git --no-pager diff-index --relative -z -M HEAD -- . in
>> background... done 0.42 seconds
>> Running git --no-pager ls-files -z -o --directory --no-empty-directory
>> --exclude-standard -- . in background... done 0.10 seconds
>> 
>> In Emacs 26.2, a "vc-dir" command runs the following:
>> 
>> Running git --no-pager update-index --refresh . in background... done
>> 0.46 seconds
>> Running git --no-pager diff-index --relative -z -M HEAD -- . in
>> background... done 0.46 seconds
>> Running git --no-pager ls-files -z -c -s -- . in background... done 4.49 seconds
>> Running git --no-pager ls-files -z -o --directory --no-empty-directory
>> --exclude-standard -- . in background... done 0.09 seconds
>> 
>> The extra command is "git --no-pager ls-files -z -c -s -- ." and it takes 4.5
>> seconds to run (I ran each test multiple times and the times are consistent).
>> However, when I run that command in a terminal, it finishes in less than a
>> second.
> 
> But the times you cited originally were much longer: 2 min.  What
> caused such a stark difference, from 2 min to just 4.5 sec?

The repository at my work place has more than 50000 files and vc-dir takes 2 minutes (measured using a stopwatch) for that repo. 

These tests I did at home against the Emacs repository which had approximately 3000 files. Originally I said this takes about 7 seconds, but that was “measured” by counting slowly in my head, these latest numbers actually measure the exact time. 

> 
>> An Emacs-Lisp program to run this script, but ignore all output, it takes 16
>> seconds, in both 25.2 and 26.1:
> 
> Once again, 16 sec is much shorter than 2 min you originally cited.

The ls-files command from for my work repo generates 6.5 Mb of data, my test program only generated 1 Mb of data.

> 
>> It seems reading output from a process in Emacs is somehow limited to 64 Kb /
>> second -- I verified this with procmon.  It looks like the behavior is the
>> same in 25.2 and 26.1, it is just that in vc-dir 25.2 does not run a git
>> command that produces a lot of output.
> 
> Does it help to set w32-pipe-buffer-size to some large value, like
> 16384, and/or decrease w32-pipe-read-delay?

I will try that tonight. Thanks for the suggestion. 

Alex

> 
> Thanks.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On Fri, Jun 22, 2018 at 4:57 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Alex Harsanyi <alexharsanyi@gmail.com>

>> It seems reading output from a process in Emacs is somehow limited to 64 Kb /
>> second -- I verified this with procmon.  It looks like the behavior is the
>> same in 25.2 and 26.1, it is just that in vc-dir 25.2 does not run a git
>> command that produces a lot of output.
>
> Does it help to set w32-pipe-buffer-size to some large value, like
> 16384, and/or decrease w32-pipe-read-delay?

If I set w32-pipe-read-delay to 0 the speed improvement is
significant.  On top of that, there is an additional speed
improvement if I set w32-pipe-buffer-size to 16384, I will use
both these values in my init file.  Thanks for helping me diagnose
this.

The default values on my system were 50 msec for the read delay
and 0 for the buffer size, which according to the doc string it
defaults to 4096.  The limits the maximum read speed from a pipe
to 80Kbps (probably less, since the 50 msec is the sleep time and
actual reading the data would take some time as well). These
values would be a significant impact on reading git ls-files
output.

Out of curiosity, Eli, since you mentioned that there is no
performance impact for vc-dir on your system, what values do you
use for these two variables?

Alex.


>
> Thanks.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Date: Fri, 22 Jun 2018 20:46:59 +0800
> Cc: emacs-devel@gnu.org
> 
> > Does it help to set w32-pipe-buffer-size to some large value, like
> > 16384, and/or decrease w32-pipe-read-delay?
> 
> If I set w32-pipe-read-delay to 0 the speed improvement is
> significant.  On top of that, there is an additional speed
> improvement if I set w32-pipe-buffer-size to 16384

Can you show numbers for these speedups, and compare the times with
Emacs 25.  It would also be useful if you show times with different
values of these, so that the optimal values could be chosen.

I'd be also interested in the timings with the larger repository.

> I will use both these values in my init file.

I recommend against using non-default values globally, as it will bite
you elsewhere.  Emacs has many different use cases wrt running async
subprocesses, and in most of them, the amount of produced output is
quite small, or is received in relatively small chunks.  The defaults
are set to provide reasonable performance and responsiveness in those
case, and shouldn't be changed just because this one use case needs
different values.

What we should do instead is bind these to non-default values in the
vc-git command that produces a lot of output that needs to be read as
quickly as possible.

> Out of curiosity, Eli, since you mentioned that there is no
> performance impact for vc-dir on your system, what values do you
> use for these two variables?

I didn't change the defaults, and the above should explain why.

Thanks.

w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1)

[Stefan Monnier]

BTW, it seems that w32-pipe-read-delay aims to solve a similar issue as
process-adaptive-read-buffering.

Does process-adaptive-read-buffering currently apply to w32 subprocesses?
If so, could it be that w32-pipe-read-delay was made obsolete when
process-adaptive-read-buffering was added?  If not, why not
(i.e. what's the fundamental difference between the two)?


        Stefan

Re: w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 22 Jun 2018 16:43:40 -0400
> 
> BTW, it seems that w32-pipe-read-delay aims to solve a similar issue as
> process-adaptive-read-buffering.

Similar, but different.  It is different in 2 important aspects:

  . w32-pipe-read-delay affects reading only from pipes, whereas
    process-adaptive-read-buffering affects any kind of subprocess;
  . w32-pipe-read-delay is about the sub-process writing to the pipe,
    whereas process-adaptive-read-buffering is about Emacs reading the
    data, and is generally related to how low-level I/O stuff works on
    the system where Emacs runs.

> Does process-adaptive-read-buffering currently apply to w32 subprocesses?

Yes, of course.  But I have never seen it being used or useful on
Windows.

> If so, could it be that w32-pipe-read-delay was made obsolete when
> process-adaptive-read-buffering was added?  If not, why not
> (i.e. what's the fundamental difference between the two)?

I think we should set w32-pipe-read-delay to zero by default
regardless.  As the comment in w32.c, where it is used, explains, it
was added to cater to invoking DOS programs on Windows 9X, which is by
now an extremely rare situation, to say the least.  I just tried the
"dir" command, which it also mentions, with and without the delay, in
a large directory which produces 650KB of output, and saw now effect
at all.

It may still be necessary to set the value non-zero for when the
sub-process treats the pipe as a console, which might be true for some
Cygwin/MSYS ports.  So I'd wait with obsoleting this variable until we
collect enough (negative) experience after changing the value.

(I'm surprised that this variable had such a profound effect with
Git's ls-files, but maybe that has something to do with the MSYS
components in Git on Windows.)

Fwd: w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1)

[Alex Harsanyi]

On Sat, Jun 23, 2018 at 3:21 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> (I'm surprised that this variable had such a profound effect with
> Git's ls-files, but maybe that has something to do with the MSYS
> components in Git on Windows.)
>

I don't think this has anything to do with Git or MSYS: I wrote a C++ program
to output 1 Mb of data (similar to the perl program I wrote earlier) and I
compiled this using Visual Studio, to get a native windows console
application.  I than set w32-pipe-buffer-size to 0 and experimented with
various settings for the w32-pipe-read-delay, using the `time-process`
function (see my previous email).  Here are the results:

* when w32-pipe-read-delay is 50 milliseconds, it takes 13.2 seconds to read 1
  Mb of piped data from the C++ executable.

* when it is 25 milliseconds, it takes 6.8 secons to read the same data

* when it is 12 milliseconds, it takes 3.47 seconds to read the same data

* when it is 0, it takes 0.13 seconds

Increasing w32-pipe-buffer-size had no effect on the speed -- I know I said in
a previous email that it had an effect, but I only tried to increase it after
dropping w32-pipe-read-delay to 0 and tried it once -- sometimes the time
drops further from 0.13 to 0.8 seconds, but this seems random -- at this
speed, probably other factors, such as process startup, have an effect on the
read speed.

On Monday, I will test it against my work repository with 50000 files, and
will report back on that.

Alex.

Re: w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1)

[Alex Harsanyi]

On Sat, Jun 23, 2018 at 9:30 PM, Alex Harsanyi <alexharsanyi@gmail.com> wrote:

> Increasing w32-pipe-buffer-size had no effect on the speed -- I know I said in
> a previous email that it had an effect, but I only tried to increase it after
> dropping w32-pipe-read-delay to 0 and tried it once -- sometimes the time
> drops further from 0.13 to 0.8 seconds, but this seems random -- at this
> speed, probably other factors, such as process startup, have an effect on the
> read speed.

Oops, above I meant to say 0.08, not 0.8, here some sequential runs with
`w32-pipe-read-delay` set to 0 and `w32-pipe-buffer-size` set to 0:

    time-process: it took 0.08 seconds
    time-process: it took 0.14 seconds
    time-process: it took 0.09 seconds
    time-process: it took 0.14 seconds [2 times]
    time-process: it took 0.14 seconds
    time-process: it took 0.09 seconds
    time-process: it took 0.14 seconds
    time-process: it took 0.11 seconds
    time-process: it took 0.13 seconds
    time-process: it took 0.09 seconds
    time-process: it took 0.14 seconds
    time-process: it took 0.09 seconds

Re: w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1)

[Eli Zaretskii]

> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Date: Sat, 23 Jun 2018 21:32:24 +0800
> 
> Oops, above I meant to say 0.08, not 0.8, here some sequential runs with
> `w32-pipe-read-delay` set to 0 and `w32-pipe-buffer-size` set to 0:
> 
>     time-process: it took 0.08 seconds
>     time-process: it took 0.14 seconds
>     time-process: it took 0.09 seconds
>     time-process: it took 0.14 seconds [2 times]
>     time-process: it took 0.14 seconds
>     time-process: it took 0.09 seconds
>     time-process: it took 0.14 seconds
>     time-process: it took 0.11 seconds
>     time-process: it took 0.13 seconds
>     time-process: it took 0.09 seconds
>     time-process: it took 0.14 seconds
>     time-process: it took 0.09 seconds

I'm confused: w32-pipe-buffer-size is zero by default, so what can we
learn about it from the above experiment?

Re: w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1)

[Alex Harsanyi]

On 23 Jun 2018, at 9:40 pm, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>> Date: Sat, 23 Jun 2018 21:32:24 +0800
>> 
>> Oops, above I meant to say 0.08, not 0.8, here some sequential runs with
>> `w32-pipe-read-delay` set to 0 and `w32-pipe-buffer-size` set to 0:
>> 
>>    time-process: it took 0.08 seconds
>>    time-process: it took 0.14 seconds
>>    time-process: it took 0.09 seconds
>>    time-process: it took 0.14 seconds [2 times]
>>    time-process: it took 0.14 seconds
>>    time-process: it took 0.09 seconds
>>    time-process: it took 0.14 seconds
>>    time-process: it took 0.11 seconds
>>    time-process: it took 0.13 seconds
>>    time-process: it took 0.09 seconds
>>    time-process: it took 0.14 seconds
>>    time-process: it took 0.09 seconds
> 
> I'm confused: w32-pipe-buffer-size is zero by default, so what can we
> learn about it from the above experiment?

Increasing w32-pipe-buffer-size has no effect on the speed of reading from the pipe. I tried several buffer size values, in combination with several read delay values, but only the read delay has an effect. For example, at 50 milliseconds read delay, it takes ~13 seconds to read 1 Mb of data, regardless if the buffer size is set to 0, 4096, 8192 or 16384. 

In one of my previous emails I indicated that increasing the buffer size made things faster, but that was an incorrect test, since I only run the test once, and there is some variation in the timings — this is what I tried to explain in the last email. 

Apologies for the confusion. 

Alex. 

Re: w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1)

[Eli Zaretskii]

> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Date: Sat, 23 Jun 2018 22:00:39 +0800
> Cc: emacs-devel@gnu.org
> 
> Increasing w32-pipe-buffer-size has no effect on the speed of reading from the pipe. I tried several buffer size values, in combination with several read delay values, but only the read delay has an effect. For example, at 50 milliseconds read delay, it takes ~13 seconds to read 1 Mb of data, regardless if the buffer size is set to 0, 4096, 8192 or 16384. 

OK, thanks for clearing this out.  It sounds like thee default value
of w32-pipe-buffer-size is doing fine, and w32-pipe-read-delay should
be reset to zero, which will probably happen in a future version of
Emacs anyway.

Re: w32-pipe-read-delay (was: vc-dir operation is very slow on large git repositories in Emacs 26.1)

[Eli Zaretskii]

> Date: Sat, 23 Jun 2018 17:31:06 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Alex Harsanyi <alexharsanyi@gmail.com>
> > Date: Sat, 23 Jun 2018 22:00:39 +0800
> > Cc: emacs-devel@gnu.org
> > 
> > Increasing w32-pipe-buffer-size has no effect on the speed of reading from the pipe. I tried several buffer size values, in combination with several read delay values, but only the read delay has an effect. For example, at 50 milliseconds read delay, it takes ~13 seconds to read 1 Mb of data, regardless if the buffer size is set to 0, 4096, 8192 or 16384. 
> 
> OK, thanks for clearing this out.  It sounds like thee default value
> of w32-pipe-buffer-size is doing fine, and w32-pipe-read-delay should
> be reset to zero, which will probably happen in a future version of
> Emacs anyway.

Done on master.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On Sat, Jun 23, 2018 at 2:24 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>> Date: Fri, 22 Jun 2018 20:46:59 +0800
>> Cc: emacs-devel@gnu.org
>>
>> > Does it help to set w32-pipe-buffer-size to some large value, like
>> > 16384, and/or decrease w32-pipe-read-delay?
>>
>> If I set w32-pipe-read-delay to 0 the speed improvement is
>> significant.  On top of that, there is an additional speed
>> improvement if I set w32-pipe-buffer-size to 16384
>
> Can you show numbers for these speedups, and compare the times with
> Emacs 25.  It would also be useful if you show times with different
> values of these, so that the optimal values could be chosen.
>
> I'd be also interested in the timings with the larger repository.

I tested various values for `w32-pipe-read-delay` on my work git repository.
The original timings that I made were against a work tree that had several
modified files and I used a stop watch to time the update.  These tests were
run on a large (51678 tracked files) but clean working directory, and I
modified the source files to time the commands using `current-time`, so I got
more precise timings.

    $ git ls-files . | wc -l
    51678

    $ git status
    On branch ah_tmp
    nothing to commit, working directory clean

Emacs 26.1 runs the following commands as part of a `vc-dir` command, or an
update of the `*vc-dir*` buffer (Emacs 25.2 runs the same commands, except it
does not run the third one)

    * git --no-pager update-index --refresh .
    * git --no-pager diff-index --relative -z -M HEAD -- .
    * git --no-pager ls-files -z -c -s -- .
    * git --no-pager ls-files -z -o --directory --no-empty-directory
--exclude-standard -- .

On a clean working directory, only "git --no-pager ls-files -z -c -s -- ."
produces any output, but produces 6.3 Mb of it.  The command itself runs fast:

    $ git --no-pager ls-files -z -c -s -- . | wc
          0  184463 6689336

    $ time git --no-pager ls-files -z -c -s -- . > /dev/null

    real    0m0.619s
    user    0m0.000s
    sys     0m0.015s

I tried several values for `w32-pipe-read-delay` to see how long it takes to
run these commands in Emacs 26.1:

`w32-pipe-read-delay` is set to 50 milliseconds (default for Emacs 26.1)

    Running git --no-pager update-index --refresh . in background...
done 0.26 seconds
    Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.34 seconds
    Running git --no-pager ls-files -z -c -s -- . in background...
done 83.70 seconds
    Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.64 seconds

`w32-pipe-read-delay` set to 25 milliseconds

    Running git --no-pager update-index --refresh . in background...
done 0.26 seconds
    Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.34 seconds
    Running git --no-pager ls-files -z -c -s -- . in background...
done 42.77 seconds
    Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.67 seconds

`w32-pipe-read-delay` set to 12 milliseconds

    Running git --no-pager update-index --refresh . in background...
done 0.27 seconds
    Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.35 seconds
    Running git --no-pager ls-files -z -c -s -- . in background...
done 21.57 seconds
    Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.65 seconds

`w32-pipe-read-delay` set to 0 milliseconds (no delay)

    Running git --no-pager update-index --refresh . in background...
done 0.31 seconds
    Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.39 seconds
    Running git --no-pager ls-files -z -c -s -- . in background...
done 1.07 seconds
    Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.69 seconds

So, with a 50 millisecond delay, it takes 83.7 seconds to read the command
output and this is reduced to 1.07 seconds when the delay is set to 0.

Alex.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Bastian Beischer]

Hi,

On Mon, Jun 25, 2018 at 12:40 PM, Alex Harsanyi <alexharsanyi@gmail.com> wrote:
>
> On Sat, Jun 23, 2018 at 2:24 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Alex Harsanyi <alexharsanyi@gmail.com>
> >> Date: Fri, 22 Jun 2018 20:46:59 +0800
> >> Cc: emacs-devel@gnu.org
> >>
> >> > Does it help to set w32-pipe-buffer-size to some large value, like
> >> > 16384, and/or decrease w32-pipe-read-delay?
> >>
> >> If I set w32-pipe-read-delay to 0 the speed improvement is
> >> significant.  On top of that, there is an additional speed
> >> improvement if I set w32-pipe-buffer-size to 16384
> >
> > Can you show numbers for these speedups, and compare the times with
> > Emacs 25.  It would also be useful if you show times with different
> > values of these, so that the optimal values could be chosen.
> >
> > I'd be also interested in the timings with the larger repository.
>
> I tested various values for `w32-pipe-read-delay` on my work git repository.
> The original timings that I made were against a work tree that had several
> modified files and I used a stop watch to time the update.  These tests were
> run on a large (51678 tracked files) but clean working directory, and I
> modified the source files to time the commands using `current-time`, so I got
> more precise timings.
>
>     $ git ls-files . | wc -l
>     51678
>
>     $ git status
>     On branch ah_tmp
>     nothing to commit, working directory clean
>
> Emacs 26.1 runs the following commands as part of a `vc-dir` command, or an
> update of the `*vc-dir*` buffer (Emacs 25.2 runs the same commands, except it
> does not run the third one)
>
>     * git --no-pager update-index --refresh .
>     * git --no-pager diff-index --relative -z -M HEAD -- .
>     * git --no-pager ls-files -z -c -s -- .
>     * git --no-pager ls-files -z -o --directory --no-empty-directory
> --exclude-standard -- .
>
> On a clean working directory, only "git --no-pager ls-files -z -c -s -- ."
> produces any output, but produces 6.3 Mb of it.  The command itself runs fast:
>
>     $ git --no-pager ls-files -z -c -s -- . | wc
>           0  184463 6689336
>
>     $ time git --no-pager ls-files -z -c -s -- . > /dev/null
>
>     real    0m0.619s
>     user    0m0.000s
>     sys     0m0.015s
>
> I tried several values for `w32-pipe-read-delay` to see how long it takes to
> run these commands in Emacs 26.1:
>
> `w32-pipe-read-delay` is set to 50 milliseconds (default for Emacs 26.1)
>
>     Running git --no-pager update-index --refresh . in background...
> done 0.26 seconds
>     Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.34 seconds
>     Running git --no-pager ls-files -z -c -s -- . in background...
> done 83.70 seconds
>     Running git --no-pager ls-files -z -o --directory
> --no-empty-directory --exclude-standard -- . in background... done
> 0.64 seconds
>
> `w32-pipe-read-delay` set to 25 milliseconds
>
>     Running git --no-pager update-index --refresh . in background...
> done 0.26 seconds
>     Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.34 seconds
>     Running git --no-pager ls-files -z -c -s -- . in background...
> done 42.77 seconds
>     Running git --no-pager ls-files -z -o --directory
> --no-empty-directory --exclude-standard -- . in background... done
> 0.67 seconds
>
> `w32-pipe-read-delay` set to 12 milliseconds
>
>     Running git --no-pager update-index --refresh . in background...
> done 0.27 seconds
>     Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.35 seconds
>     Running git --no-pager ls-files -z -c -s -- . in background...
> done 21.57 seconds
>     Running git --no-pager ls-files -z -o --directory
> --no-empty-directory --exclude-standard -- . in background... done
> 0.65 seconds
>
> `w32-pipe-read-delay` set to 0 milliseconds (no delay)
>
>     Running git --no-pager update-index --refresh . in background...
> done 0.31 seconds
>     Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.39 seconds
>     Running git --no-pager ls-files -z -c -s -- . in background...
> done 1.07 seconds
>     Running git --no-pager ls-files -z -o --directory
> --no-empty-directory --exclude-standard -- . in background... done
> 0.69 seconds
>
> So, with a 50 millisecond delay, it takes 83.7 seconds to read the command
> output and this is reduced to 1.07 seconds when the delay is set to 0.

From these timings it appears that you have been using a 4096 byte
buffer. The logic is as follows:

Assume that all the delay in the 83.7 seconds is due to 50ms waits,
one such wait whenever the buffer is full. This is based on the fact
that you observe a ~proportional slowdown when increasing the
read-delay. Then this means that the buffer was full 1674 times, which
means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096
bytes. So the predicted approximate delay for a message of size S is

D = (S / B) * W

where B is the buffer size (4096) and W is the w32-pipe-read-delay
(0.05 s). Based on that logic it appears that setting
w32-pipe-buffer-size to 16384 should give you a ~four times smaller
total delay - especially when w32-pipe-read-delay is 50 ms, for
example. Unless the OS refuses to provide you with such a buffer size?

>
>
> Alex.
>

Cheers
Bastian

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On Mon, Jun 25, 2018 at 7:09 PM, Bastian Beischer
<bastian.beischer@rwth-aachen.de> wrote:
> Hi,
>
> On Mon, Jun 25, 2018 at 12:40 PM, Alex Harsanyi <alexharsanyi@gmail.com> wrote:

>> So, with a 50 millisecond delay, it takes 83.7 seconds to read the command
>> output and this is reduced to 1.07 seconds when the delay is set to 0.
>
> From these timings it appears that you have been using a 4096 byte
> buffer. The logic is as follows:
>
> Assume that all the delay in the 83.7 seconds is due to 50ms waits,
> one such wait whenever the buffer is full. This is based on the fact
> that you observe a ~proportional slowdown when increasing the
> read-delay. Then this means that the buffer was full 1674 times, which
> means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096
> bytes. So the predicted approximate delay for a message of size S is
>
> D = (S / B) * W
>
> where B is the buffer size (4096) and W is the w32-pipe-read-delay
> (0.05 s). Based on that logic it appears that setting
> w32-pipe-buffer-size to 16384 should give you a ~four times smaller
> total delay - especially when w32-pipe-read-delay is 50 ms, for
> example. Unless the OS refuses to provide you with such a buffer size?
>

Setting w32-pipe-buffer-size has no effect on the timings, see my
previous emails about this.   I do not know why.

Alex.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> From: Bastian Beischer <bastian.beischer@rwth-aachen.de>
> Date: Mon, 25 Jun 2018 13:09:37 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-Devel <emacs-devel@gnu.org>
> 
> >From these timings it appears that you have been using a 4096 byte
> buffer.

Yes, the default buffer size of a pipe on Windows is 4KB.

> Assume that all the delay in the 83.7 seconds is due to 50ms waits,
> one such wait whenever the buffer is full. This is based on the fact
> that you observe a ~proportional slowdown when increasing the
> read-delay. Then this means that the buffer was full 1674 times, which
> means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096
> bytes. So the predicted approximate delay for a message of size S is
> 
> D = (S / B) * W
> 
> where B is the buffer size (4096) and W is the w32-pipe-read-delay
> (0.05 s). Based on that logic it appears that setting
> w32-pipe-buffer-size to 16384 should give you a ~four times smaller
> total delay - especially when w32-pipe-read-delay is 50 ms, for
> example. Unless the OS refuses to provide you with such a buffer size?

I think you ignore a few factors here.  Are you familiar with how
Emacs on Windows reads data from async subprocesses?  If not, I
suggest to read the large comment around line 800 of w32proc.c and
study the code it describes.

With 50-msec delay, it is reasonable to assume that the pipe's buffer
gets filled while the reader thread waits for 50 msec, so most of the
time is indeed due to these waits.  But with zero waits, its quite
possible that Emacs reads the stuff from the pipe as fast as Git
writes to its other end, because the reading can begin very soon after
the first byte is written to the pipe (in reality, writes are
probably buffered, so the first chunk is much larger than 1 byte).  If
Emacs empties the pipe as fast as Git writes to it or faster, the size
of the pipe will have (almost) no effect whatsoever.

> Unless the OS refuses to provide you with such a buffer size?

No, there should be no such problem.

Enlarging the pipe buffer size is (IME) only needed when Emacs needs
to _write_ large amounts of data to the subprocesses, and the
subprocesses doesn't keep up.  We write to the pipe in the main
thread, so with small enough buffer and slow enough reading by the
subprocesses will eventually cause Emacs to hang, which is not really
user-friendly.  See bug#22344 and bug#24143, for example.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Bastian Beischer]

On Mon, Jun 25, 2018 at 5:17 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Bastian Beischer <bastian.beischer@rwth-aachen.de>
>> Date: Mon, 25 Jun 2018 13:09:37 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-Devel <emacs-devel@gnu.org>
>>
>> >From these timings it appears that you have been using a 4096 byte
>> buffer.
>
> Yes, the default buffer size of a pipe on Windows is 4KB.
>
>> Assume that all the delay in the 83.7 seconds is due to 50ms waits,
>> one such wait whenever the buffer is full. This is based on the fact
>> that you observe a ~proportional slowdown when increasing the
>> read-delay. Then this means that the buffer was full 1674 times, which
>> means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096
>> bytes. So the predicted approximate delay for a message of size S is
>>
>> D = (S / B) * W
>>
>> where B is the buffer size (4096) and W is the w32-pipe-read-delay
>> (0.05 s). Based on that logic it appears that setting
>> w32-pipe-buffer-size to 16384 should give you a ~four times smaller
>> total delay - especially when w32-pipe-read-delay is 50 ms, for
>> example. Unless the OS refuses to provide you with such a buffer size?
>
> I think you ignore a few factors here.

Yes indeed, but the fact that one can go to ~1s by reducing the delay
suggests that all of this can be ignored to first order. It looks like
the input rate to the pipe can be considered infinite in a good
approximation.

> Are you familiar with how
> Emacs on Windows reads data from async subprocesses?  If not, I
> suggest to read the large comment around line 800 of w32proc.c and
> study the code it describes.

No I'm not familiar with the actual code - you are very right to point
me to it and I will study it. I'm just making some very naive
observations :)

>
> With 50-msec delay, it is reasonable to assume that the pipe's buffer
> gets filled while the reader thread waits for 50 msec, so most of the
> time is indeed due to these waits.  But with zero waits, its quite
> possible that Emacs reads the stuff from the pipe as fast as Git
> writes to its other end, because the reading can begin very soon after
> the first byte is written to the pipe (in reality, writes are
> probably buffered, so the first chunk is much larger than 1 byte).  If
> Emacs empties the pipe as fast as Git writes to it or faster, the size
> of the pipe will have (almost) no effect whatsoever.

I agree with all of this, but the point is that _if_ with 50 msec
delay the buffer get's filled completely, we should assume that a
buffer twice as large woud also get filled completely, provided that
the input rate is near infinite. I think that assumption is valid
because the total time is linear with the delay down to delays of
order 5 msec at least. That should result in a data rate which is
twice as large. Therefore I don't understand how the timing can be
independent of the buffer size. But maybe studying the code will make
me understand the situation :)

Looking at commit 58a622d473112f8ff5b4bdb3e49bc6573dfd3404 (where you
introduced w32-pipe-buffer-size) I can see a code path that results in
resetting the buffer size to 4 KB. Maybe that get's taken?

>
>> Unless the OS refuses to provide you with such a buffer size?
>
> No, there should be no such problem.
>
> Enlarging the pipe buffer size is (IME) only needed when Emacs needs
> to _write_ large amounts of data to the subprocesses, and the
> subprocesses doesn't keep up.  We write to the pipe in the main
> thread, so with small enough buffer and slow enough reading by the
> subprocesses will eventually cause Emacs to hang, which is not really
> user-friendly.  See bug#22344 and bug#24143, for example.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> From: Bastian Beischer <bastian.beischer@rwth-aachen.de>
> Date: Mon, 25 Jun 2018 17:44:21 +0200
> Cc: Alex Harsanyi <alexharsanyi@gmail.com>, Emacs-Devel <emacs-devel@gnu.org>
> 
> I agree with all of this, but the point is that _if_ with 50 msec
> delay the buffer get's filled completely, we should assume that a
> buffer twice as large woud also get filled completely, provided that
> the input rate is near infinite. I think that assumption is valid
> because the total time is linear with the delay down to delays of
> order 5 msec at least. That should result in a data rate which is
> twice as large. Therefore I don't understand how the timing can be
> independent of the buffer size.

For small enough buffer sizes (32KB etc.), 50 msec is an eternity, I
think.  AFAIR, modern versions of Windows read/write data in 64KB
chunks.

> Looking at commit 58a622d473112f8ff5b4bdb3e49bc6573dfd3404 (where you
> introduced w32-pipe-buffer-size) I can see a code path that results in
> resetting the buffer size to 4 KB. Maybe that get's taken?

Only if the GetNamedPipeInfo call fails, which I don't expect to
happen (and should be easy to verify by running Emacs under a
debugger, where the debug message in that case should be visible).

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On Tue, Jun 26, 2018 at 12:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Bastian Beischer <bastian.beischer@rwth-aachen.de>

>
> > Looking at commit 58a622d473112f8ff5b4bdb3e49bc6573dfd3404 (where you
> > introduced w32-pipe-buffer-size) I can see a code path that results in
> > resetting the buffer size to 4 KB. Maybe that get's taken?
>
> Only if the GetNamedPipeInfo call fails, which I don't expect to
> happen (and should be easy to verify by running Emacs under a
> debugger, where the debug message in that case should be visible).

Setting w32-pipe-buffer-size to a large value has no effect on the
reading speed even if w32-pipe-read-delay is set to 50 milliseconds.
I mentioned this in one of my previous emails.

Alex.


Alex.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Date: Mon, 25 Jun 2018 18:40:08 +0800
> Cc: emacs-devel@gnu.org
> 
> So, with a 50 millisecond delay, it takes 83.7 seconds to read the command
> output and this is reduced to 1.07 seconds when the delay is set to 0.

Thanks for the data, I think the conclusion is clear.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/20/18 6:51 PM, Eli Zaretskii wrote:

> FWIW, I get about 2 sec for this in the Emacs repository, and I see no
> perceptible difference between Emacs 25.2 and Emacs 26.1.

I have also noticed that the refresh operation in vc-dir buffer became 
slower (but the projects I work with are of moderate side, so it's 
liveable).

I wrote about it here: 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25683#44

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, Tom Tromey <tom@tromey.com>
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 22 Jun 2018 12:04:16 +0300
> 
> I have also noticed that the refresh operation in vc-dir buffer became 
> slower (but the projects I work with are of moderate side, so it's 
> liveable).
> 
> I wrote about it here: 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25683#44

Since we added a command that produces a lot of output, some slowdown
is to be expected, no?

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/23/18 4:37 PM, Eli Zaretskii wrote:

> Since we added a command that produces a lot of output, some slowdown
> is to be expected, no?

Right. So my concern here is whether we really need to use such a 
command. Or, rather, how much work is it going to be to avoid using it.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> Cc: alexharsanyi@gmail.com, tom@tromey.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 23 Jun 2018 21:28:14 +0300
> 
> On 6/23/18 4:37 PM, Eli Zaretskii wrote:
> 
> > Since we added a command that produces a lot of output, some slowdown
> > is to be expected, no?
> 
> Right. So my concern here is whether we really need to use such a 
> command. Or, rather, how much work is it going to be to avoid using it.

I assume it was added to fix some problem or issue.  What was that
issue, and what are the alternatives that avoid issuing the command?

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/23/18 9:44 PM, Eli Zaretskii wrote:

> I assume it was added to fix some problem or issue.  What was that
> issue, and what are the alternatives that avoid issuing the command?

It's used to obtain the list of conflicted files. Which is the new 
feature added in the bug report I've referenced.

I'm not quite sure why we have to fetch the list of all files in the 
repository to do that.

It seems like the approach was simply adapted from the 
ls-files-up-to-date stage, but that one is only called for a list of 
individual files, not for the status of a whole directory (e.g. whole 
repository).

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> Cc: alexharsanyi@gmail.com, tom@tromey.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 25 Jun 2018 15:55:38 +0300
> 
> It's used to obtain the list of conflicted files. Which is the new 
> feature added in the bug report I've referenced.
> 
> I'm not quite sure why we have to fetch the list of all files in the 
> repository to do that.
> 
> It seems like the approach was simply adapted from the 
> ls-files-up-to-date stage, but that one is only called for a list of 
> individual files, not for the status of a whole directory (e.g. whole 
> repository).

Maybe some Git expert could suggest a more economical way of doing the
same?  (I'm not that expert.)

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: alexharsanyi@gmail.com, tom@tromey.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Mon, 25 Jun 2018 15:55:38 +0300
>> 
>> It's used to obtain the list of conflicted files. Which is the new 
>> feature added in the bug report I've referenced.
>> 
>> I'm not quite sure why we have to fetch the list of all files in the 
>> repository to do that.
>> 
>> It seems like the approach was simply adapted from the 
>> ls-files-up-to-date stage, but that one is only called for a list of 
>> individual files, not for the status of a whole directory (e.g. whole 
>> repository).
>
> Maybe some Git expert could suggest a more economical way of doing the
> same?  (I'm not that expert.)

Not exactly an expert, but googling suggests 'git ls-files -u' as the
answer. Please be aware that the output will have 3 entries for every
conflicting file: merge base, theirs, and ours.

-- Sergey

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/26/18 6:14 PM, Sergey Organov wrote:

> Not exactly an expert, but googling suggests 'git ls-files -u' as the
> answer. Please be aware that the output will have 3 entries for every
> conflicting file: merge base, theirs, and ours.

Tom, any reason you did not choose this option?

In my brief testing, it works the same. Only faster.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Tom Tromey]

>> Not exactly an expert, but googling suggests 'git ls-files -u' as the
>> answer. Please be aware that the output will have 3 entries for every
>> conflicting file: merge base, theirs, and ours.

Dmitry> Tom, any reason you did not choose this option?

I'm afraid I don't remember any more.

Tom

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/27/18 5:49 AM, Tom Tromey wrote:
>>> Not exactly an expert, but googling suggests 'git ls-files -u' as the
>>> answer. Please be aware that the output will have 3 entries for every
>>> conflicting file: merge base, theirs, and ours.
> 
> Dmitry> Tom, any reason you did not choose this option?
> 
> I'm afraid I don't remember any more.

OK. Too bad the commit didn't include tests.

So I think we should try it and see if anything broken comes up?

Eli, OK for emacs-26?

diff --git a/lisp/vc/vc-git.el b/lisp/vc/vc-git.el
index 88052d7749..3f66bff89f 100644
--- a/lisp/vc/vc-git.el
+++ b/lisp/vc/vc-git.el
@@ -584,7 +584,7 @@ vc-git-dir-status-goto-stage
                         "ls-files" "-z" "-c" "-s" "--"))
        (`ls-files-conflict
         (vc-git-command (current-buffer) 'async files
-                       "ls-files" "-z" "-c" "-s" "--"))
+                       "ls-files" "-z" "-u" "-s" "--"))
        (`ls-files-unknown
         (vc-git-command (current-buffer) 'async files
                         "ls-files" "-z" "-o" "--directory"

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, Sergey Organov <sorganov@gmail.com>,
>  alexharsanyi@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 27 Jun 2018 12:41:49 +0300
> 
> So I think we should try it and see if anything broken comes up?

Yes.  Although I'm slightly scared because the man pages say

  OPTIONS
    -c
    --cached
      Show cached files in the output (default)
  ...
    -u
    --unmerged
      Show unmerged files in the output (forces --stage)

which doesn't sound equivalent to me.  But then I can never understand
the Git documentation well enough to be sure what it says...

> Eli, OK for emacs-26?

If you are sure -c and -u are equivalent here, yes, please.

Thanks.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/27/18 6:12 PM, Eli Zaretskii wrote:

>> So I think we should try it and see if anything broken comes up?
> 
> Yes.  Although I'm slightly scared because the man pages say
> 
>    OPTIONS
>      -c
>      --cached
>        Show cached files in the output (default)
>    ...
>      -u
>      --unmerged
>        Show unmerged files in the output (forces --stage)
> 
> which doesn't sound equivalent to me.  But then I can never understand
> the Git documentation well enough to be sure what it says...

We don't need equivalent. The first option returns *all* files in the 
repository, which we want to get away from.

>> Eli, OK for emacs-26?
> 
> If you are sure -c and -u are equivalent here, yes, please.

It works in my testing. But if I were 100% sure, I wouldn't be asking.

Pushed, with a couple more tweaks (orthogonal ones).

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> Cc: alexharsanyi@gmail.com, tom@tromey.com, sorganov@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 28 Jun 2018 03:30:36 +0300
> 
> >    OPTIONS
> >      -c
> >      --cached
> >        Show cached files in the output (default)
> >    ...
> >      -u
> >      --unmerged
> >        Show unmerged files in the output (forces --stage)
> > 
> > which doesn't sound equivalent to me.  But then I can never understand
> > the Git documentation well enough to be sure what it says...
> 
> We don't need equivalent. The first option returns *all* files in the 
> repository, which we want to get away from.

That wasn't my concern.  My concern was about "cached" vs "unmerged".
I think I understand the former (it's about the staged files, right?),
but what does the latter mean, and how does it affect what we need in
this case from Git.  IOW, if -c is not necessary, why not just disable
it?  Or is that not possible, and -u is the way to disable it?

> Pushed, with a couple more tweaks (orthogonal ones).

Thanks.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/28/18 3:55 PM, Eli Zaretskii wrote:

> That wasn't my concern.  My concern was about "cached" vs "unmerged".
> I think I understand the former (it's about the staged files, right?),
> but what does the latter mean, and how does it affect what we need in
> this case from Git.  IOW, if -c is not necessary, why not just disable
> it?  Or is that not possible, and -u is the way to disable it?

-c and -u are mutually exclusive options.

"cached", here, means all registered (or simply staged) files.

-u limits the output to unmerged files only.

So the question here is whether "unmerged" covers all files we want to 
assign the "conflict" status to. I think it does.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> Cc: alexharsanyi@gmail.com, tom@tromey.com, sorganov@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 28 Jun 2018 16:06:27 +0300
> 
> -u limits the output to unmerged files only.
> 
> So the question here is whether "unmerged" covers all files we want to 
> assign the "conflict" status to. I think it does.

Yes, and I still don't think I understand what "unmerged" means in
this context.  Can you explain?  To me, "merging" only has meaning
when more than one branch is considered, or two instances of a single
branch that diverged.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/28/18 4:12 PM, Eli Zaretskii wrote:

> Yes, and I still don't think I understand what "unmerged" means in
> this context.  Can you explain?  To me, "merging" only has meaning
> when more than one branch is considered, or two instances of a single
> branch that diverged.

And what does the "conflict" status mean to you?

 From the vc-state docstring:

   `conflict'       The file contains conflicts as the result of a merge.
                    For now the conflicts are text conflicts.  In the
                    future this might be extended to deal with metadata
                    conflicts too.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> Cc: alexharsanyi@gmail.com, tom@tromey.com, sorganov@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 28 Jun 2018 16:18:52 +0300
> 
> On 6/28/18 4:12 PM, Eli Zaretskii wrote:
> 
> > Yes, and I still don't think I understand what "unmerged" means in
> > this context.  Can you explain?  To me, "merging" only has meaning
> > when more than one branch is considered, or two instances of a single
> > branch that diverged.
> 
> And what does the "conflict" status mean to you?

You are saying that "unmerged" alludes to files that failed to merge,
i.e. "in conflict" in our parlance?  If so, fine with me.  I just
wonder why didn't Git documentation say so.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/28/18 4:38 PM, Eli Zaretskii wrote:

> You are saying that "unmerged" alludes to files that failed to merge,
> i.e. "in conflict" in our parlance?  If so, fine with me.  I just
> wonder why didn't Git documentation say so.

That's their terminology, and it's also used in the command line UI:

$ git status
On branch master
You have unmerged paths.
   (fix conflicts and run "git commit")

...

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 28 Jun 2018 16:47:29 +0300
> Cc: alexharsanyi@gmail.com, tom@tromey.com, sorganov@gmail.com,
> 	emacs-devel@gnu.org
> 
> On 6/28/18 4:38 PM, Eli Zaretskii wrote:
> 
> > You are saying that "unmerged" alludes to files that failed to merge,
> > i.e. "in conflict" in our parlance?  If so, fine with me.  I just
> > wonder why didn't Git documentation say so.
> 
> That's their terminology

<Shrug> not according to the Git Glossary.

Thanks.

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Sergey Organov]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 6/27/18 5:49 AM, Tom Tromey wrote:
>>>> Not exactly an expert, but googling suggests 'git ls-files -u' as the
>>>> answer. Please be aware that the output will have 3 entries for every
>>>> conflicting file: merge base, theirs, and ours.
>>
>> Dmitry> Tom, any reason you did not choose this option?
>>
>> I'm afraid I don't remember any more.
>
> OK. Too bad the commit didn't include tests.
>
> So I think we should try it and see if anything broken comes up?

In fact it could be that

$ git status -z

is the way to go, even though it will need changes in parsing.

Here is quote from 'git help status', for quick reference:

       -z
           Terminate entries with NUL, instead of LF. This implies the
           --porcelain output format if no other format is given.
[...]

       --porcelain
           Give the output in an easy-to-parse format for scripts. This is
           similar to the short output, but will remain stable across Git
           versions and regardless of user configuration. See below for
           details.


The output format is then carefully described later in the man-page.
 
-- Sergey

Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Dmitry Gutov]

On 6/28/18 4:21 PM, Sergey Organov wrote:

> In fact it could be that
> 
> $ git status -z
> 
> is the way to go, even though it will need changes in parsing.
> 
> Here is quote from 'git help status', for quick reference:
> 
>         -z
>             Terminate entries with NUL, instead of LF. This implies the
>             --porcelain output format if no other format is given.
> [...]
> 
>         --porcelain
>             Give the output in an easy-to-parse format for scripts. This is
>             similar to the short output, but will remain stable across Git
>             versions and regardless of user configuration. See below for
>             details.
> 
> 
> The output format is then carefully described later in the man-page.

Switching over to 'git status porcelain=v2' is a bigger enterprise, and 
we only want to do that after Git 2.11.0 (where version 2 was 
introduced) is old enough.