find-file takes forever on slow network file systems and very deep directories

find-file takes forever on slow network file systems and very deep directories

[Andy L]

Here are details: http://tinyurl.com/4o6q49

A.

find-file takes forever on slow network file systems and very deep directories

[Andy Leszczynski]

Here are details: http://tinyurl.com/4o6q49

A.

Re: find-file takes forever on slow network file systems and very deep directories

[Eli Zaretskii]

> Date: Mon, 22 Sep 2008 20:36:35 -0700
> From: Andy Leszczynski <andy.leszczynski@gmail.com>
> 
> Here are details: http://tinyurl.com/4o6q49

Thanks.  Is it possible to add a trace from Emacs 21.3?  That'd give
us a clue where to look for the differences.

Re: find-file takes forever on slow network file systems and very deep directories

[Eli Zaretskii]

> Date: Tue, 23 Sep 2008 11:14:06 -0700
> From: "Andrzej Leszczynski" <andy.leszczynski@gmail.com>
> 
> >> Here are details: http://tinyurl.com/4o6q49
> >
> > Thanks.  Is it possible to add a trace from Emacs 21.3?  That'd give
> > us a clue where to look for the differences.
> >
> 
> see attached (done on GNU Emacs 21.3.1 (i386-mingw-nt5.1.2600) of
> 2004-03-10 on NYAUMO)

Thanks.  I cannot spot anything significant: the traces look almost
identical, and the code of the suspect function (emulated `stat') is
virtually the same, up until the point where the value of
w32-get-true-file-attributes is checked (the value of which you said
made no difference).

Previously you said that with Emacs 22.x, the number of file ops is
889, but I don't see so many ops in the trace you posted to the above
URL.  What am I missing?

I also don't see the 15-sec delay in the trace: both the trace from
Emacs 22.x and from 21.x show about 2-3 seconds between the first
syscall and the last.

Please help me understand the data.

Re: find-file takes forever on slow network file systems and very deep directories

[Eli Zaretskii]

[Please keep the emacs-devel list on the list of addressees.]

> Date: Tue, 23 Sep 2008 12:51:23 -0700
> From: "Andrzej Leszczynski" <andy.leszczynski@gmail.com>
> 
> > Thanks.  I cannot spot anything significant: the traces look almost
> > identical, and the code of the suspect function (emulated `stat') is
> > virtually the same, up until the point where the value of
> > w32-get-true-file-attributes is checked (the value of which you said
> > made no difference).
> >
> > Previously you said that with Emacs 22.x, the number of file ops is
> > 889, but I don't see so many ops in the trace you posted to the above
> > URL.  What am I missing?
> >
> > I also don't see the 15-sec delay in the trace: both the trace from
> > Emacs 22.x and from 21.x show about 2-3 seconds between the first
> > syscall and the last.
> >
> > Please help me understand the data.
> >
> 
> the original log embedded in the email contains almost 900 lines and
> take 15 seconds - I guess you need to click on "read more"
> 
> 64      2:45:51 PM
> [...]
> 952     2:46:06 PM
> 
> and it is on emacs run with no .emacs file. Plase doble check if you
> can see that.  w32-get-true-file-attributes does not improve it.

Right, I see the whole 900+ lines now.

It looks like the VC package could be the culprit: there are many
filesystem calls that look for files such as:

  W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\.bzr\checkout\
  W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\RCS\ 
  W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\CVS\Entries
  W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\.svn\entries
  W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\SCCS\
  W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\file.c\_MTN\
  W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\{arch}\
  W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\file.c\MCVS\

This greatly enlarges the number of files and directories Emacs needs
to `stat'.

Can you try evaluating the following expression (in Emacs 22.x):

  (remove-hook 'find-file-hook 'vc-find-file-hook)

and then see if the speed of `find-file' increases significantly as a
result?

Re: find-file takes forever on slow network file systems and very deep directories

[Eli Zaretskii]

> Date: Thu, 25 Sep 2008 09:26:59 -0700
> From: "Andrzej Leszczynski" <andy.leszczynski@gmail.com>
> 
> > It looks like the VC package could be the culprit: there are many
> > filesystem calls that look for files such as:
> > 
> >   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\.bzr\checkout\
> >   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\RCS\ 
> >   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\CVS\Entries
> >   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\.svn\entries
> >   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\SCCS\
> >   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\file.c\_MTN\
> >   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\{arch}\
> >   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\file.c\MCVS\
> >
> > This greatly enlarges the number of files and directories Emacs needs
> > to `stat'.
> >
> > Can you try evaluating the following expression (in Emacs 22.x):
> >
> >  (remove-hook 'find-file-hook 'vc-find-file-hook)
> >
> > and then see if the speed of `find-file' increases significantly as a
> > result?
> >
> 
> I had the same thought - see in the USENET thread - but I did not have
> the clue how to disable it. Now it  loads the file much faster,
> although it still _stats_ all the dirs up to the root (drive C:)

Does anyone have any thoughts on how to avoid such a significant
performance hit (15 sec for a simple `find-file') on Samba volumes, as
opposed to just 2 seconds in Emacs 21.x, which didn't automatically
have `vc-find-file-hook' on the `find-file-hook' list?

If there's no good cure, we should at least provide a user option to
avoid putting `vc-find-file-hook' on `find-file-hook', or maybe allow
finer-grain control of it, like per filesystem.

Thoughts?

Re: find-file takes forever on slow network file systems and very deep directories

[Lennart Borgman (gmail)]

Eli Zaretskii wrote:
>> Date: Thu, 25 Sep 2008 09:26:59 -0700
>> From: "Andrzej Leszczynski" <andy.leszczynski@gmail.com>
>>
>>> It looks like the VC package could be the culprit: there are many
>>> filesystem calls that look for files such as:
>>>
>>>   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\.bzr\checkout\
>>>   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\RCS\ 
>>>   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\CVS\Entries
>>>   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\.svn\entries
>>>   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\SCCS\
>>>   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\file.c\_MTN\
>>>   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\{arch}\
>>>   W:\dir1\dir2\dir3\dir4\dir5\dir6\dir7\dir8\dir9\dirA\dirB\dirC\file.c\MCVS\
>>>
>>> This greatly enlarges the number of files and directories Emacs needs
>>> to `stat'.
>>>
>>> Can you try evaluating the following expression (in Emacs 22.x):
>>>
>>>  (remove-hook 'find-file-hook 'vc-find-file-hook)
>>>
>>> and then see if the speed of `find-file' increases significantly as a
>>> result?
>>>
>> I had the same thought - see in the USENET thread - but I did not have
>> the clue how to disable it. Now it  loads the file much faster,
>> although it still _stats_ all the dirs up to the root (drive C:)
> 
> Does anyone have any thoughts on how to avoid such a significant
> performance hit (15 sec for a simple `find-file') on Samba volumes, as
> opposed to just 2 seconds in Emacs 21.x, which didn't automatically
> have `vc-find-file-hook' on the `find-file-hook' list?
> 
> If there's no good cure, we should at least provide a user option to
> avoid putting `vc-find-file-hook' on `find-file-hook', or maybe allow
> finer-grain control of it, like per filesystem.
> 
> Thoughts?


I took a quick look at `vc-find-file-hook' and it looks like it is
caching the `backend' property for the file name. Should not the check
for the vc backend happen only once for each dir (like dir1, dir2 etc)
above?

Perhaps there is a bug somewhere so that the caching is not used?

Re: find-file takes forever on slow network file systems and very deep directories

[Stefan Monnier]

> Does anyone have any thoughts on how to avoid such a significant
> performance hit (15 sec for a simple `find-file') on Samba volumes, as
> opposed to just 2 seconds in Emacs 21.x, which didn't automatically
> have `vc-find-file-hook' on the `find-file-hook' list?

> If there's no good cure, we should at least provide a user option to
> avoid putting `vc-find-file-hook' on `find-file-hook', or maybe allow
> finer-grain control of it, like per filesystem.

AFAIK, the same happened under Emacs-21.  The only difference is the
number of backends we check, as well as the fact that most/all the new
backends need to look at parent-dirs whereas the older backends only
need to look at the current dir.

Currently, there are 2 ways to control this problem:
1 - Trim vc-handled-backends so it only includes backends about which
    you care.
2 - Set vc-ignore-dir-regexp to turn off VC in your Samba subtree.


-- Stefan

Re: find-file takes forever on slow network file systems and very deep directories

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org,  Andrzej Leszczynski <andy.leszczynski@gmail.com>
> Date: Fri, 26 Sep 2008 11:05:51 -0400
> 
> > Does anyone have any thoughts on how to avoid such a significant
> > performance hit (15 sec for a simple `find-file') on Samba volumes, as
> > opposed to just 2 seconds in Emacs 21.x, which didn't automatically
> > have `vc-find-file-hook' on the `find-file-hook' list?
> 
> > If there's no good cure, we should at least provide a user option to
> > avoid putting `vc-find-file-hook' on `find-file-hook', or maybe allow
> > finer-grain control of it, like per filesystem.
> 
> AFAIK, the same happened under Emacs-21.

Not with "emacs -Q", AFAICS.

> 2 - Set vc-ignore-dir-regexp to turn off VC in your Samba subtree.

That'd be the solution I was looking for.  Thanks.

Andy, does this do what you want?

(I still have on my TODO to make most callers of `stat' on Windows
bind w32-get-true-file-attributes to nil, which will make find-file
even faster.)

Re: find-file takes forever on slow network file systems and very deep directories

[Stefan Monnier]

>> > Does anyone have any thoughts on how to avoid such a significant
>> > performance hit (15 sec for a simple `find-file') on Samba volumes, as
>> > opposed to just 2 seconds in Emacs 21.x, which didn't automatically
>> > have `vc-find-file-hook' on the `find-file-hook' list?
>> 
>> > If there's no good cure, we should at least provide a user option to
>> > avoid putting `vc-find-file-hook' on `find-file-hook', or maybe allow
>> > finer-grain control of it, like per filesystem.
>> AFAIK, the same happened under Emacs-21.
> Not with "emacs -Q", AFAICS.

IIRC Emacs-21 only supported CVS, RCS, and SCCS, neither of which needs
to walk up the directory tree, so the behavior was indeed different.


        Stefan