all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#24126: vc-hg-state can be extremely slow
@ 2016-08-01 18:40 Jonathan Kotta
  2016-08-02 13:32 ` Dmitry Gutov
  0 siblings, 1 reply; 7+ messages in thread
From: Jonathan Kotta @ 2016-08-01 18:40 UTC (permalink / raw)
  To: 24126

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Emacs uses `hg status -A` in vc-hg-state, which in turn is used in many vc
commands (e.g. vc-root-diff).  The "-A" option makes mercurial look at all
files under the directory, even the ignored ones.  If there are a lot of
ignored files, this will be very slow.

As an example, I have a repo that's 38MB / 300 files when freshly checked
out, and 34GB / 1.2M files when the build finishes (if you're curious, it's
a yocto project).  Without clearing the disk cache, `hg stat -A >
/dev/null` takes 28s; it's far longer if the disk cache isn't warmed up or
the output is actually used.  `hg status` takes about 90ms.

vg-git-state does not have this problem; currently it behaves like `hg
status`, i.e. honoring the ignore rules.   There is actually a FIXME
comment regarding this functionality, noting that `git ls-files -i -o
--exclude-standard` is the equivalent to `hg status -A`; this takes over
400s (I got sick of waiting).

I'm guessing VC has some sort of assumption that vc-x-state will return all
files.  Maybe the command could bailout after taking too long and use `hg
status`.  Maybe the command options could be configurable.  Personally, I'd
prefer just dropping the "-A", because I've never used it and I don't
really see why you'd want to get ignored files by default; this is my
current solution.

I'm using Emacs 24.5.1.  I've tested Emacs 25.1.1 and it still has the
issue; though vc-hg-state has changed, it still uses "-A" and is still very
slow on my repo.

-- 
Thanks,

Jonathan Kotta

Hofstadter's Law:
    It always takes longer than you expect, even
    when you take into account Hofstadter's Law.

[-- Attachment #2: Type: text/html, Size: 1945 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#24126: vc-hg-state can be extremely slow
  2016-08-01 18:40 bug#24126: vc-hg-state can be extremely slow Jonathan Kotta
@ 2016-08-02 13:32 ` Dmitry Gutov
  2016-08-02 15:57   ` Jonathan Kotta
  0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Gutov @ 2016-08-02 13:32 UTC (permalink / raw)
  To: Jonathan Kotta, 24126

On 08/01/2016 09:40 PM, Jonathan Kotta wrote:
> Emacs uses `hg status -A` in vc-hg-state, which in turn is used in many
> vc commands (e.g. vc-root-diff).  The "-A" option makes mercurial look
> at all files under the directory, even the ignored ones.  If there are a
> lot of ignored files, this will be very slow.

Why does it do that? We're passing a specific file name to it.

> As an example, I have a repo that's 38MB / 300 files when freshly
> checked out, and 34GB / 1.2M files when the build finishes (if you're
> curious, it's a yocto project).  Without clearing the disk cache, `hg
> stat -A > /dev/null` takes 28s;

What about 'hg status -A file/name > /dev/null'?

> vg-git-state does not have this problem; currently it behaves like `hg
> status`, i.e. honoring the ignore rules.   There is actually a FIXME
> comment regarding this functionality, noting that `git ls-files -i -o
> --exclude-standard` is the equivalent to `hg status -A`; this takes over
> 400s (I got sick of waiting).

The FIXME is outdated, we'll do it by parsing 'git status --porcelain'.

Does 'git status --ignored --porcelain -- file/name' take a lot of time 
for you as well?

> I'm guessing VC has some sort of assumption that vc-x-state will return
> all files.

vc-x-state returns the state of a single file.





^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#24126: vc-hg-state can be extremely slow
  2016-08-02 13:32 ` Dmitry Gutov
@ 2016-08-02 15:57   ` Jonathan Kotta
  2016-08-02 16:19     ` Dmitry Gutov
  0 siblings, 1 reply; 7+ messages in thread
From: Jonathan Kotta @ 2016-08-02 15:57 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: 24126

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

On Tue, Aug 2, 2016 at 8:32 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 08/01/2016 09:40 PM, Jonathan Kotta wrote:
>
>> Emacs uses `hg status -A` in vc-hg-state, which in turn is used in many
>> vc commands (e.g. vc-root-diff).  The "-A" option makes mercurial look
>> at all files under the directory, even the ignored ones.  If there are a
>> lot of ignored files, this will be very slow.
>>
>
> Why does it do that? We're passing a specific file name to it.
>

In the case of `vc-root-diff` at least, it's passing in a directory; the
command is essentially `hg status -A ./`, and the CWD is indeed the repo
root.  I did this by running `vc-root-diff` from a dired buffer visiting
the repo root.

-- 
Thanks,

Jonathan Kotta

Hofstadter's Law:
    It always takes longer than you expect, even
    when you take into account Hofstadter's Law.

[-- Attachment #2: Type: text/html, Size: 1474 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#24126: vc-hg-state can be extremely slow
  2016-08-02 15:57   ` Jonathan Kotta
@ 2016-08-02 16:19     ` Dmitry Gutov
  2021-08-15 12:30       ` Lars Ingebrigtsen
  0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Gutov @ 2016-08-02 16:19 UTC (permalink / raw)
  To: Jonathan Kotta; +Cc: 24126

On 08/02/2016 06:57 PM, Jonathan Kotta wrote:

> In the case of `vc-root-diff` at least, it's passing in a directory; the
> command is essentially `hg status -A ./`, and the CWD is indeed the repo
> root.  I did this by running `vc-root-diff` from a dired buffer visiting
> the repo root.

OK, thanks. The chain of calls looks like this:

   vc-hg-state("~/vc/mozilla-central/")
   vc-hg-registered("~/vc/mozilla-central/")
   apply(vc-hg-registered "~/vc/mozilla-central/")
   vc-call-backend(Hg registered "~/vc/mozilla-central/")
   vc-registered("~/vc/mozilla-central/")
   vc-backend("~/vc/mozilla-central/")
   vc-working-revision("~/vc/mozilla-central/")
   vc-root-diff(nil t)

Maybe vc-hg-registered shouldn't delegate to vc-hg-state, and call 'hg 
status' on its own without '-A'.





^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#24126: vc-hg-state can be extremely slow
  2016-08-02 16:19     ` Dmitry Gutov
@ 2021-08-15 12:30       ` Lars Ingebrigtsen
  2021-08-17  0:14         ` Dmitry Gutov
  0 siblings, 1 reply; 7+ messages in thread
From: Lars Ingebrigtsen @ 2021-08-15 12:30 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: 24126, Jonathan Kotta

Dmitry Gutov <dgutov@yandex.ru> writes:

>> In the case of `vc-root-diff` at least, it's passing in a directory; the
>> command is essentially `hg status -A ./`, and the CWD is indeed the repo
>> root.  I did this by running `vc-root-diff` from a dired buffer visiting
>> the repo root.
>
> OK, thanks. The chain of calls looks like this:
>
>   vc-hg-state("~/vc/mozilla-central/")
>   vc-hg-registered("~/vc/mozilla-central/")
>   apply(vc-hg-registered "~/vc/mozilla-central/")
>   vc-call-backend(Hg registered "~/vc/mozilla-central/")
>   vc-registered("~/vc/mozilla-central/")
>   vc-backend("~/vc/mozilla-central/")
>   vc-working-revision("~/vc/mozilla-central/")
>   vc-root-diff(nil t)

This was five years ago -- I tried reproducing this by instrumenting
`vg-hg-state' in a hg-covered directory, and that function was not
called when doing vc-root-diff there.

So has this been fixed in the intervening years?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#24126: vc-hg-state can be extremely slow
  2021-08-15 12:30       ` Lars Ingebrigtsen
@ 2021-08-17  0:14         ` Dmitry Gutov
  2021-08-18 14:39           ` Lars Ingebrigtsen
  0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Gutov @ 2021-08-17  0:14 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: 24126, Jonathan Kotta

On 15.08.2021 15:30, Lars Ingebrigtsen wrote:
> This was five years ago -- I tried reproducing this by instrumenting
> `vg-hg-state' in a hg-covered directory, and that function was not
> called when doing vc-root-diff there.

I think that only happens once per root per Emacs session (after that 
the directory's backend is cached).

> So has this been fixed in the intervening years?

Probably not.

It's not too slow here in the project I've tried it with, but it doesn't 
have many ignored files (which was the situation in the report).





^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#24126: vc-hg-state can be extremely slow
  2021-08-17  0:14         ` Dmitry Gutov
@ 2021-08-18 14:39           ` Lars Ingebrigtsen
  0 siblings, 0 replies; 7+ messages in thread
From: Lars Ingebrigtsen @ 2021-08-18 14:39 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: 24126, Jonathan Kotta

Dmitry Gutov <dgutov@yandex.ru> writes:

> I think that only happens once per root per Emacs session (after that
> the directory's backend is cached).

Yup.  If I instrument the function and then do a vc-root-diff in a fresh
Emacs session, I get the expected backtrace:

Debugger entered--entering a function:
* vc-hg-state("/tmp/hg/")
  apply(vc-hg-state "/tmp/hg/")
  vc-call-backend(Hg state "/tmp/hg/")
  vc-state-refresh("/tmp/hg/" Hg)
  vc-state("/tmp/hg/" Hg)
  vc-hg-registered("/tmp/hg/")
  apply(vc-hg-registered "/tmp/hg/")
  vc-call-backend(Hg registered "/tmp/hg/")
  #f(compiled-function (b) #<bytecode -0x1056748a7cc3b17d>)(Hg)
  mapc(#f(compiled-function (b) #<bytecode -0x1056748a7cc3b17d>) (RCS CVS SVN SCCS SRC Bzr Git Hg))
  vc-registered("/tmp/hg/")
  vc-backend("/tmp/hg/")
  vc-working-revision("/tmp/hg/")
  vc-root-diff(nil)
  eval((vc-root-diff nil) t)
  eval-expression((vc-root-diff nil) nil nil 127)
  funcall-interactively(eval-expression (vc-root-diff nil) nil nil 127)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-08-18 14:39 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-08-01 18:40 bug#24126: vc-hg-state can be extremely slow Jonathan Kotta
2016-08-02 13:32 ` Dmitry Gutov
2016-08-02 15:57   ` Jonathan Kotta
2016-08-02 16:19     ` Dmitry Gutov
2021-08-15 12:30       ` Lars Ingebrigtsen
2021-08-17  0:14         ` Dmitry Gutov
2021-08-18 14:39           ` Lars Ingebrigtsen

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.