23.0.60; Slowdown in directory scanning over time.

23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

[Resending this, because I'm not sure if my email made it through the
first time]

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the emacs-pretest-bug@gnu.org mailing list.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

I'm using CVS emacs and I've noticed that it's exhibiting a very
painful slowdown over time.  I'm primarily using JDEE for Java
development, and when opening a new java file it can get to taking
minutes for the file to open.  If I restart emacs things are snappy
again.  I have tried to narrow things down and think I have found the
problem.  When opening a java file, JDEE does a scan of library
directories to build the project classpath, and it seems to be this
scanning that is taking longer to evaluate.  When switching to an
already open Java buffer, it does a quick scan looking in various
directories for changes to the project file, and this also slows down.

If I evaluate the following code:

  (let ((time-initial (current-time))
        (files (directory-files "/home2/len/reeltwo_sandboxes/vetech/src/com/vetech/datamodel")))
    (message (format "Took %d seconds to list %d files" (time-to-seconds (time-subtract (current-time) time-initial)) (length files))))

with the emacs I just had running (which had been open for several
days, but only had approximately 35 buffers open), the output said it
took between 2 and 3 seconds to list 166 files.  With this freshly
restarted emacs, it now repeatedly says 0 seconds.  I hope you can pin
things down to find the actual cause.


Cheers,
Len.


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/home2/len/local/share/emacs/23.0.60/etc/DEBUG for instructions.


In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.10.4)
 of 2008-03-25 on noir
configured using `configure  '--with-x-toolkit=gtk' '--prefix=/home2/len/local''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_NZ
  value of $XMODIFIERS: @im=none
  locale-coding-system: iso-latin-1-unix
  default-enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  shell-dirtrack-mode: t
  pmwiki-minor-mode: t
  erc-autoaway-mode: t
  erc-button-mode: t
  erc-ring-mode: t
  erc-track-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-pcomplete-mode: t
  display-time-mode: t
  delete-selection-mode: t
  show-paren-mode: t
  pc-selection-mode: t
  tool-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
ESC x r i c DEL DEL DEL i r c RET j e TAB r TAB DEL 
DEL TAB h t t p : / / w w w . j w z . o r g / d o c 
/ t a b s - v s - s p a c e s . h t m l RET k u r TAB 
h t t p : / / w w w . j w z . o r g / d o c / t a b 
s - v s - s p a c e s . h t m l RET C-x C-f ~ / t e 
TAB . e l TAB RET C-n C-n C-n C-n C-b C-x C-e C-w C-x 
C-e C-x C-e C-x C-e C-x C-e C-n ESC x r e p o r TAB 
b u TAB RET

Recent messages:
Took 0 seconds to list 166 files
"Took 0 seconds to list 166 files"
kill-region: The mark is not set now, so there is no region
Took 0 seconds to list 166 files
"Took 0 seconds to list 166 files"
Took 0 seconds to list 166 files
"Took 0 seconds to list 166 files"
Took 0 seconds to list 166 files
"Took 0 seconds to list 166 files"
Took 0 seconds to list 166 files
"Took 0 seconds to list 166 files"

Re: 23.0.60; Slowdown in directory scanning over time.

[Lennart Borgman (gmail)]

JDEE and the things it depends on are not yet a part of Emacs, 
unfortunately. I don't know exactly where this message should go, but I 
guess that cedet-devel and jdee-devel both might be good places so I 
have CC:ed them.


Len Trigg wrote:
> [Resending this, because I'm not sure if my email made it through the
> first time]
> 
> Please write in English if possible, because the Emacs maintainers
> usually do not have translators to read other languages for them.
> 
> Your bug report will be posted to the emacs-pretest-bug@gnu.org mailing list.
> 
> Please describe exactly what actions triggered the bug
> and the precise symptoms of the bug:
> 
> I'm using CVS emacs and I've noticed that it's exhibiting a very
> painful slowdown over time.  I'm primarily using JDEE for Java
> development, and when opening a new java file it can get to taking
> minutes for the file to open.  If I restart emacs things are snappy
> again.  I have tried to narrow things down and think I have found the
> problem.  When opening a java file, JDEE does a scan of library
> directories to build the project classpath, and it seems to be this
> scanning that is taking longer to evaluate.  When switching to an
> already open Java buffer, it does a quick scan looking in various
> directories for changes to the project file, and this also slows down.
> 
> If I evaluate the following code:
> 
>   (let ((time-initial (current-time))
>         (files (directory-files "/home2/len/reeltwo_sandboxes/vetech/src/com/vetech/datamodel")))
>     (message (format "Took %d seconds to list %d files" (time-to-seconds (time-subtract (current-time) time-initial)) (length files))))
> 
> with the emacs I just had running (which had been open for several
> days, but only had approximately 35 buffers open), the output said it
> took between 2 and 3 seconds to list 166 files.  With this freshly
> restarted emacs, it now repeatedly says 0 seconds.  I hope you can pin
> things down to find the actual cause.
> 
> 
> Cheers,
> Len.
> 
> 
> If Emacs crashed, and you have the Emacs process in the gdb debugger,
> please include the output from the following gdb commands:
>     `bt full' and `xbacktrace'.
> If you would like to further debug the crash, please read the file
> /home2/len/local/share/emacs/23.0.60/etc/DEBUG for instructions.
> 
> 
> In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.10.4)
>  of 2008-03-25 on noir
> configured using `configure  '--with-x-toolkit=gtk' '--prefix=/home2/len/local''
> 
> Important settings:
>   value of $LC_ALL: nil
>   value of $LC_COLLATE: nil
>   value of $LC_CTYPE: nil
>   value of $LC_MESSAGES: nil
>   value of $LC_MONETARY: nil
>   value of $LC_NUMERIC: nil
>   value of $LC_TIME: nil
>   value of $LANG: en_NZ
>   value of $XMODIFIERS: @im=none
>   locale-coding-system: iso-latin-1-unix
>   default-enable-multibyte-characters: t
> 
> Major mode: Emacs-Lisp
> 
> Minor modes in effect:
>   shell-dirtrack-mode: t
>   pmwiki-minor-mode: t
>   erc-autoaway-mode: t
>   erc-button-mode: t
>   erc-ring-mode: t
>   erc-track-mode: t
>   erc-fill-mode: t
>   erc-stamp-mode: t
>   erc-pcomplete-mode: t
>   display-time-mode: t
>   delete-selection-mode: t
>   show-paren-mode: t
>   pc-selection-mode: t
>   tool-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   global-auto-composition-mode: t
>   auto-composition-mode: t
>   auto-compression-mode: t
>   column-number-mode: t
>   line-number-mode: t
>   transient-mark-mode: t
> 
> Recent input:
> ESC x r i c DEL DEL DEL i r c RET j e TAB r TAB DEL 
> DEL TAB h t t p : / / w w w . j w z . o r g / d o c 
> / t a b s - v s - s p a c e s . h t m l RET k u r TAB 
> h t t p : / / w w w . j w z . o r g / d o c / t a b 
> s - v s - s p a c e s . h t m l RET C-x C-f ~ / t e 
> TAB . e l TAB RET C-n C-n C-n C-n C-b C-x C-e C-w C-x 
> C-e C-x C-e C-x C-e C-x C-e C-n ESC x r e p o r TAB 
> b u TAB RET
> 
> Recent messages:
> Took 0 seconds to list 166 files
> "Took 0 seconds to list 166 files"
> kill-region: The mark is not set now, so there is no region
> Took 0 seconds to list 166 files
> "Took 0 seconds to list 166 files"
> Took 0 seconds to list 166 files
> "Took 0 seconds to list 166 files"
> Took 0 seconds to list 166 files
> "Took 0 seconds to list 166 files"
> Took 0 seconds to list 166 files
> "Took 0 seconds to list 166 files"
> 
> 
> 

Re: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

Lennart Borgman wrote:
> JDEE and the things it depends on are not yet a part of Emacs,
> unfortunately. I don't know exactly where this message should go, but
> I guess that cedet-devel and jdee-devel both might be good places so I
> have CC:ed them.

I had already narrowed the source of the problem down to the
'directory-files' function (see the example sexp I included that
demonstrates the problem), which is built in.  I think that lets JDEE
and cedet off the hook.

Cheers,
Len.

Re: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

Len Trigg wrote:
> If I evaluate the following code:
> 
>   (let ((time-initial (current-time))
>         (files (directory-files "/home2/len/reeltwo_sandboxes/vetech/src/com/vetech/datamodel")))
>     (message (format "Took %d seconds to list %d files" (time-to-seconds (time-subtract (current-time) time-initial)) (length files))))
> 
> with the emacs I just had running (which had been open for several
> days, but only had approximately 35 buffers open), the output said it
> took between 2 and 3 seconds to list 166 files.  With this freshly
> restarted emacs, it now repeatedly says 0 seconds.  I hope you can pin
> things down to find the actual cause.

Can someone please try to replicate this problem by trying the above
snippet in an emacs that has been running for several days, for a
directory containing many files?  Is this not causing problems for
other people (maybe emacs developers tend not to have long-running
emacs)?  Previously I could have an emacs running for months without
restarting, but now I have to restart it every couple of days because
it gets too slow.

Cheers,
Len.