bug#24126: vc-hg-state can be extremely slow

bug#24126: vc-hg-state can be extremely slow

[Jonathan Kotta]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Emacs uses `hg status -A` in vc-hg-state, which in turn is used in many vc
commands (e.g. vc-root-diff).  The "-A" option makes mercurial look at all
files under the directory, even the ignored ones.  If there are a lot of
ignored files, this will be very slow.

As an example, I have a repo that's 38MB / 300 files when freshly checked
out, and 34GB / 1.2M files when the build finishes (if you're curious, it's
a yocto project).  Without clearing the disk cache, `hg stat -A >
/dev/null` takes 28s; it's far longer if the disk cache isn't warmed up or
the output is actually used.  `hg status` takes about 90ms.

vg-git-state does not have this problem; currently it behaves like `hg
status`, i.e. honoring the ignore rules.   There is actually a FIXME
comment regarding this functionality, noting that `git ls-files -i -o
--exclude-standard` is the equivalent to `hg status -A`; this takes over
400s (I got sick of waiting).

I'm guessing VC has some sort of assumption that vc-x-state will return all
files.  Maybe the command could bailout after taking too long and use `hg
status`.  Maybe the command options could be configurable.  Personally, I'd
prefer just dropping the "-A", because I've never used it and I don't
really see why you'd want to get ignored files by default; this is my
current solution.

I'm using Emacs 24.5.1.  I've tested Emacs 25.1.1 and it still has the
issue; though vc-hg-state has changed, it still uses "-A" and is still very
slow on my repo.

-- 
Thanks,

Jonathan Kotta

Hofstadter's Law:
    It always takes longer than you expect, even
    when you take into account Hofstadter's Law.

[-- Attachment #2: Type: text/html, Size: 1945 bytes --]

bug#24126: vc-hg-state can be extremely slow

[Dmitry Gutov]

On 08/01/2016 09:40 PM, Jonathan Kotta wrote:
> Emacs uses `hg status -A` in vc-hg-state, which in turn is used in many
> vc commands (e.g. vc-root-diff).  The "-A" option makes mercurial look
> at all files under the directory, even the ignored ones.  If there are a
> lot of ignored files, this will be very slow.

Why does it do that? We're passing a specific file name to it.

> As an example, I have a repo that's 38MB / 300 files when freshly
> checked out, and 34GB / 1.2M files when the build finishes (if you're
> curious, it's a yocto project).  Without clearing the disk cache, `hg
> stat -A > /dev/null` takes 28s;

What about 'hg status -A file/name > /dev/null'?

> vg-git-state does not have this problem; currently it behaves like `hg
> status`, i.e. honoring the ignore rules.   There is actually a FIXME
> comment regarding this functionality, noting that `git ls-files -i -o
> --exclude-standard` is the equivalent to `hg status -A`; this takes over
> 400s (I got sick of waiting).

The FIXME is outdated, we'll do it by parsing 'git status --porcelain'.

Does 'git status --ignored --porcelain -- file/name' take a lot of time 
for you as well?

> I'm guessing VC has some sort of assumption that vc-x-state will return
> all files.

vc-x-state returns the state of a single file.

bug#24126: vc-hg-state can be extremely slow

[Jonathan Kotta]

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

On Tue, Aug 2, 2016 at 8:32 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 08/01/2016 09:40 PM, Jonathan Kotta wrote:
>
>> Emacs uses `hg status -A` in vc-hg-state, which in turn is used in many
>> vc commands (e.g. vc-root-diff).  The "-A" option makes mercurial look
>> at all files under the directory, even the ignored ones.  If there are a
>> lot of ignored files, this will be very slow.
>>
>
> Why does it do that? We're passing a specific file name to it.
>

In the case of `vc-root-diff` at least, it's passing in a directory; the
command is essentially `hg status -A ./`, and the CWD is indeed the repo
root.  I did this by running `vc-root-diff` from a dired buffer visiting
the repo root.

-- 
Thanks,

Jonathan Kotta

Hofstadter's Law:
    It always takes longer than you expect, even
    when you take into account Hofstadter's Law.

[-- Attachment #2: Type: text/html, Size: 1474 bytes --]

bug#24126: vc-hg-state can be extremely slow

[Dmitry Gutov]

On 08/02/2016 06:57 PM, Jonathan Kotta wrote:

> In the case of `vc-root-diff` at least, it's passing in a directory; the
> command is essentially `hg status -A ./`, and the CWD is indeed the repo
> root.  I did this by running `vc-root-diff` from a dired buffer visiting
> the repo root.

OK, thanks. The chain of calls looks like this:

   vc-hg-state("~/vc/mozilla-central/")
   vc-hg-registered("~/vc/mozilla-central/")
   apply(vc-hg-registered "~/vc/mozilla-central/")
   vc-call-backend(Hg registered "~/vc/mozilla-central/")
   vc-registered("~/vc/mozilla-central/")
   vc-backend("~/vc/mozilla-central/")
   vc-working-revision("~/vc/mozilla-central/")
   vc-root-diff(nil t)

Maybe vc-hg-registered shouldn't delegate to vc-hg-state, and call 'hg 
status' on its own without '-A'.

bug#24126: vc-hg-state can be extremely slow

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> In the case of `vc-root-diff` at least, it's passing in a directory; the
>> command is essentially `hg status -A ./`, and the CWD is indeed the repo
>> root.  I did this by running `vc-root-diff` from a dired buffer visiting
>> the repo root.
>
> OK, thanks. The chain of calls looks like this:
>
>   vc-hg-state("~/vc/mozilla-central/")
>   vc-hg-registered("~/vc/mozilla-central/")
>   apply(vc-hg-registered "~/vc/mozilla-central/")
>   vc-call-backend(Hg registered "~/vc/mozilla-central/")
>   vc-registered("~/vc/mozilla-central/")
>   vc-backend("~/vc/mozilla-central/")
>   vc-working-revision("~/vc/mozilla-central/")
>   vc-root-diff(nil t)

This was five years ago -- I tried reproducing this by instrumenting
`vg-hg-state' in a hg-covered directory, and that function was not
called when doing vc-root-diff there.

So has this been fixed in the intervening years?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24126: vc-hg-state can be extremely slow

[Dmitry Gutov]

On 15.08.2021 15:30, Lars Ingebrigtsen wrote:
> This was five years ago -- I tried reproducing this by instrumenting
> `vg-hg-state' in a hg-covered directory, and that function was not
> called when doing vc-root-diff there.

I think that only happens once per root per Emacs session (after that 
the directory's backend is cached).

> So has this been fixed in the intervening years?

Probably not.

It's not too slow here in the project I've tried it with, but it doesn't 
have many ignored files (which was the situation in the report).

bug#24126: vc-hg-state can be extremely slow

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> I think that only happens once per root per Emacs session (after that
> the directory's backend is cached).

Yup.  If I instrument the function and then do a vc-root-diff in a fresh
Emacs session, I get the expected backtrace:

Debugger entered--entering a function:
* vc-hg-state("/tmp/hg/")
  apply(vc-hg-state "/tmp/hg/")
  vc-call-backend(Hg state "/tmp/hg/")
  vc-state-refresh("/tmp/hg/" Hg)
  vc-state("/tmp/hg/" Hg)
  vc-hg-registered("/tmp/hg/")
  apply(vc-hg-registered "/tmp/hg/")
  vc-call-backend(Hg registered "/tmp/hg/")
  #f(compiled-function (b) #<bytecode -0x1056748a7cc3b17d>)(Hg)
  mapc(#f(compiled-function (b) #<bytecode -0x1056748a7cc3b17d>) (RCS CVS SVN SCCS SRC Bzr Git Hg))
  vc-registered("/tmp/hg/")
  vc-backend("/tmp/hg/")
  vc-working-revision("/tmp/hg/")
  vc-root-diff(nil)
  eval((vc-root-diff nil) t)
  eval-expression((vc-root-diff nil) nil nil 127)
  funcall-interactively(eval-expression (vc-root-diff nil) nil nil 127)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no