bug#122: 23.0.60; Slowdown in directory scanning over time.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Chong Yidong]

> I'm using CVS emacs and I've noticed that it's exhibiting a very
> painful slowdown over time.  I'm primarily using JDEE for Java
> development, and when opening a new java file it can get to taking
> minutes for the file to open.  If I restart emacs things are snappy
> again.  I have tried to narrow things down and think I have found the
> problem.  When opening a java file, JDEE does a scan of library
> directories to build the project classpath, and it seems to be this
> scanning that is taking longer to evaluate.  When switching to an
> already open Java buffer, it does a quick scan looking in various
> directories for changes to the project file, and this also slows down

I can't reproduce this.  Do you still see the problem?  If so, could you
try to make a recipe for reproducing the bug?

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

Chong Yidong wrote:
> I can't reproduce this.  Do you still see the problem?  If so, could you
> try to make a recipe for reproducing the bug?

I do still have the problem, although I have not updated my emacs
since July 7th.  I will try to update it now, and will let you know in
a couple of days time if things are degraded.  It is very easy for me
to reproduce, because it happens always :-)

Perhaps my emacs usage is unusual in some ways that cause the
degradation over time.

I typically have emacs running in one terminal window, and two other
emacsclients connected.  In terms of long-running emacs
operations/code, there is the following:

I run wanderlust as my primary email client. Running all the time.

I run erc for IRC.  Running all the time.

I have a cronjob that fetches ics calendar files, and invokes a new
emacs -eval with icalendar.el to convert to diary format, and if there
are changes, invokes emacsclient -c -eval to tell the running emacs to
refresh its diary.

I often have a bunch of Java code buffers open - this bug particularly
affects JDEE performance because it involves lots of directory
scanning when switching projects or opening new java files.


Cheers,
Len.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

Chong Yidong wrote:
> I can't reproduce this.  Do you still see the problem?  If so, could you
> try to make a recipe for reproducing the bug?

I updated my emacs to latest CVS last Wednesday (GNU Emacs 23.0.60.1
(i686-pc-linux-gnu, GTK+ Version 2.10.4) of 2008-08-27 on noir), and
evaluated the following lisp code in *scratch*:

  (let ((time-initial (current-time))
        (files (directory-files "/usr/bin"))
        (time-retrieve (current-time)))
    (insert "\n")
    (insert-date)
    (insert (format " %d files took %d seconds to list" 
                    (length files)
                    (time-to-seconds (time-subtract time-retrieve time-initial)))))

I repeated it at various times.  As you can see below, less than a
week later, there is a huge degradation. 

Mon Sep  1, 2008  9:35 AM 2471 files took 30 seconds to list
Mon Sep  1, 2008  9:34 AM 2471 files took 30 seconds to list
Sat Aug 30, 2008  9:03 AM 2471 files took 11 seconds to list
Sat Aug 30, 2008  9:02 AM 2471 files took 11 seconds to list
Thu Aug 28, 2008 11:42 AM 2471 files took 1 seconds to list
Thu Aug 28, 2008 11:35 AM 2471 files took 2 seconds to list
Wed Aug 27, 2008  3:20 PM 2471 files took 0 seconds to list

While waiting for the output, emacs is using 100% CPU.

Here is hopefully the relevant output of strace -T -r -p EMACSPID I
haven't really used the -T and -r before, but I think the first number
in the output shows relative time between invocations, and the last
number in brackets is the time spent in the system call itself, thus
it confirms the time is being spent in emacs, not system calls.

     0.000068 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 999834}}, NULL) = 0 <0.000018>
     0.000313 gettimeofday({1220219283, 414241}, NULL) = 0 <0.000015>
     0.000132 open("/usr/bin", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 14 <0.000043>
     0.000110 fstat64(14, {st_mode=S_IFDIR|0755, st_size=69632, ...}) = 0 <0.000015>
     0.000139 fcntl64(14, F_SETFD, FD_CLOEXEC) = 0 <0.000015>
     0.000080 rt_sigprocmask(SIG_BLOCK, [ALRM], [], 8) = 0 <0.000015>
     0.000087 gettimeofday({1220219283, 414785}, NULL) = 0 <0.000013>
     0.000065 gettimeofday({1220219283, 414850}, NULL) = 0 <0.000014>
     0.000067 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 998841}}, NULL) = 0 <0.000017>
     0.000084 rt_sigprocmask(SIG_UNBLOCK, [ALRM], [ALRM], 8) = 0 <0.000014>
     0.000091 getdents64(14, /* 125 entries */, 4096) = 4088 <0.000147>
     0.196466 gettimeofday({1220219283, 611650}, NULL) = 0 <0.000090>
     0.134968 gettimeofday({1220219283, 746638}, NULL) = 0 <0.000101>
     0.668723 --- SIGALRM (Alarm clock) @ 0 (0) ---
     0.000157 sigreturn()               = ? (mask now []) <0.000019>
     0.010949 rt_sigprocmask(SIG_BLOCK, [ALRM], [], 8) = 0 <0.000019>
     0.000115 gettimeofday({1220219284, 426474}, NULL) = 0 <0.000015>
     0.000076 gettimeofday({1220219284, 426549}, NULL) = 0 <0.000017>
     0.000074 rt_sigprocmask(SIG_UNBLOCK, [ALRM], [ALRM], 8) = 0 <0.000017>
     0.183893 gettimeofday({1220219284, 610605}, NULL) = 0 <0.000090>
     0.155338 gettimeofday({1220219284, 765992}, NULL) = 0 <0.000129>
     0.264379 getdents64(14, /* 125 entries */, 4096) = 4088 <0.000467>
     0.618443 gettimeofday({1220219285, 648821}, NULL) = 0 <0.000146>
     0.216144 gettimeofday({1220219285, 864973}, NULL) = 0 <0.000146>
     0.725045 getdents64(14, /* 121 entries */, 4096) = 4096 <0.000212>
     0.232641 gettimeofday({1220219286, 822618}, NULL) = 0 <0.000116>
     0.186857 gettimeofday({1220219287, 9393}, NULL) = 0 <0.000019>
     0.953455 gettimeofday({1220219287, 962841}, NULL) = 0 <0.000021>
     0.221155 gettimeofday({1220219288, 184246}, NULL) = 0 <0.000264>
     0.332566 getdents64(14, /* 122 entries */, 4096) = 4088 <0.000432>
     0.654922 gettimeofday({1220219289, 171578}, NULL) = 0 <0.000120>
     0.181435 gettimeofday({1220219289, 353032}, NULL) = 0 <0.000130>
     0.847072 getdents64(14, /* 126 entries */, 4096) = 4080 <0.000338>
     0.133808 gettimeofday({1220219290, 333910}, NULL) = 0 <0.000133>
     0.183397 gettimeofday({1220219290, 517394}, NULL) = 0 <0.000203>
     0.881111 gettimeofday({1220219291, 398399}, NULL) = 0 <0.000119>
     0.185942 gettimeofday({1220219291, 584516}, NULL) = 0 <0.000282>
     0.423669 getdents64(14, /* 125 entries */, 4096) = 4080 <0.000333>
     0.515814 gettimeofday({1220219292, 523830}, NULL) = 0 <0.000124>
     0.185801 gettimeofday({1220219292, 709673}, NULL) = 0 <0.000154>
     0.883796 gettimeofday({1220219293, 593418}, NULL) = 0 <0.000115>
     0.186351 gettimeofday({1220219293, 779819}, NULL) = 0 <0.000153>
     0.000620 getdents64(14, /* 123 entries */, 4096) = 4080 <0.000276>
     0.950282 gettimeofday({1220219294, 730672}, NULL) = 0 <0.000117>
     0.186529 gettimeofday({1220219294, 917379}, NULL) = 0 <0.000284>
     0.502483 getdents64(14, /* 128 entries */, 4096) = 4096 <0.000317>
     0.520920 gettimeofday({1220219295, 949368}, NULL) = 0 <0.008892>
     0.302211 gettimeofday({1220219296, 242849}, NULL) = 0 <0.000141>
     0.895065 gettimeofday({1220219297, 137880}, NULL) = 0 <0.000116>
     0.184786 gettimeofday({1220219297, 322849}, NULL) = 0 <0.000288>
     0.144951 getdents64(14, /* 123 entries */, 4096) = 4096 <0.000502>
     0.783568 gettimeofday({1220219298, 251184}, NULL) = 0 <0.000116>
     0.186041 gettimeofday({1220219298, 437276}, NULL) = 0 <0.000155>
     0.638823 getdents64(14, /* 126 entries */, 4096) = 4096 <0.000062>
     0.365309 gettimeofday({1220219299, 441391}, NULL) = 0 <0.000146>
     0.293914 gettimeofday({1220219299, 735296}, NULL) = 0 <0.000131>
     0.885331 gettimeofday({1220219300, 620602}, NULL) = 0 <0.000115>
     0.185862 gettimeofday({1220219300, 806530}, NULL) = 0 <0.000160>
     0.247335 getdents64(14, /* 123 entries */, 4096) = 4096 <0.000391>
     0.680369 gettimeofday({1220219301, 734169}, NULL) = 0 <0.000116>
     0.185810 gettimeofday({1220219301, 920024}, NULL) = 0 <0.000151>
     0.695677 getdents64(14, /* 125 entries */, 4096) = 4096 <0.000639>
     0.213305 gettimeofday({1220219302, 829112}, NULL) = 0 <0.000267>
     0.186323 gettimeofday({1220219303, 15327}, NULL) = 0 <0.000150>
     0.900804 gettimeofday({1220219303, 916096}, NULL) = 0 <0.000123>
     0.187633 gettimeofday({1220219304, 103903}, NULL) = 0 <0.000288>
     0.366357 getdents64(14, /* 122 entries */, 4096) = 4088 <0.000191>
     0.624951 gettimeofday({1220219305, 95037}, NULL) = 0 <0.000121>
     0.321246 gettimeofday({1220219305, 416307}, NULL) = 0 <0.000138>
     0.757553 getdents64(14, /* 123 entries */, 4096) = 4072 <0.000409>
     0.165275 gettimeofday({1220219306, 339126}, NULL) = 0 <0.000136>
     0.184346 gettimeofday({1220219306, 523504}, NULL) = 0 <0.000157>
     0.909111 gettimeofday({1220219307, 432564}, NULL) = 0 <0.000119>
     0.188846 gettimeofday({1220219307, 621582}, NULL) = 0 <0.000281>
     0.397529 getdents64(14, /* 124 entries */, 4096) = 4080 <0.000393>
     0.566455 gettimeofday({1220219308, 585390}, NULL) = 0 <0.000116>
     0.191020 gettimeofday({1220219308, 776594}, NULL) = 0 <0.000289>
     0.844134 getdents64(14, /* 125 entries */, 4096) = 4088 <0.000183>
     0.274177 gettimeofday({1220219309, 894724}, NULL) = 0 <0.000118>
     0.188975 gettimeofday({1220219310, 83747}, NULL) = 0 <0.000156>
     0.960998 gettimeofday({1220219311, 44852}, NULL) = 0 <0.000273>
     0.292174 gettimeofday({1220219311, 336893}, NULL) = 0 <0.000132>
     0.430172 getdents64(14, /* 123 entries */, 4096) = 4080 <0.006762>
     0.530705 gettimeofday({1220219312, 297744}, NULL) = 0 <0.000129>
     0.188943 gettimeofday({1220219312, 486861}, NULL) = 0 <0.000279>
     0.931694 getdents64(14, /* 127 entries */, 4096) = 4088 <0.000476>
     0.056997 gettimeofday({1220219313, 475415}, NULL) = 0 <0.000149>
     0.185205 gettimeofday({1220219313, 660754}, NULL) = 0 <0.000280>
     0.903955 gettimeofday({1220219314, 564452}, NULL) = 0 <0.000021>
     0.188058 gettimeofday({1220219314, 752530}, NULL) = 0 <0.000019>
     0.539996 getdents64(14, /* 125 entries */, 4096) = 4088 <0.000339>
     0.462233 gettimeofday({1220219315, 754826}, NULL) = 0 <0.000116>
     0.187881 gettimeofday({1220219315, 942756}, NULL) = 0 <0.000154>
     0.886213 gettimeofday({1220219316, 828929}, NULL) = 0 <0.000119>
     0.187752 gettimeofday({1220219317, 16732}, NULL) = 0 <0.000165>
     0.062955 getdents64(14, /* 110 entries */, 4096) = 3696 <0.000290>
     0.942788 gettimeofday({1220219318, 22447}, NULL) = 0 <0.000146>
     0.276459 gettimeofday({1220219318, 298898}, NULL) = 0 <0.000129>
     0.428554 getdents64(14, /* 0 entries */, 4096) = 0 <0.000156>
     0.000337 close(14)                 = 0 <0.000208>
     0.011102 gettimeofday({1220219318, 738956}, NULL) = 0 <0.000181>
     0.000579 time(NULL)                = 1220219318 <0.000101>
     0.000326 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2434, ...}) = 0 <0.000140>
     0.000397 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2434, ...}) = 0 <0.000103>
     0.000317 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2434, ...}) = 0 <0.000085>
     0.013005 gettimeofday({1220219318, 753515}, NULL) = 0 <0.000131>
     0.000311 gettimeofday({1220219318, 753755}, NULL) = 0 <0.000081>
     0.000392 gettimeofday({1220219318, 754156}, NULL) = 0 <0.000088>
     0.002620 rt_sigprocmask(SIG_BLOCK, [WINCH], [], 8) = 0 <0.000129>
     0.000295 rt_sigprocmask(SIG_BLOCK, [IO], [WINCH], 8) = 0 <0.000063>
     0.000198 rt_sigprocmask(SIG_BLOCK, [IO], [WINCH IO], 8) = 0 <0.000062>
     0.000173 ioctl(8, FIONREAD, [0])   = 0 <0.000023>
     0.000077 ioctl(6, FIONREAD, [0])   = 0 <0.000017>
     0.000065 ioctl(3, FIONREAD, [0])   = 0 <0.000016>
     0.000069 rt_sigprocmask(SIG_SETMASK, [WINCH IO], [WINCH IO], 8) = 0 <0.000014>
     0.000084 gettimeofday({1220219318, 757661}, NULL) = 0 <0.000014>
     0.000474 gettimeofday({1220219318, 758137}, NULL) = 0 <0.000015>
     0.000074 write(8, "\33[1;24r\33[15;1H\33[1L\33[1;50r\33[50;1H"..., 71) = 71 <0.000107>
 

Cheers,
Len.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Richard M. Stallman]

The way to investigate this is to run Emacs with X, under GDB, and try
stopping it during this operation by typing C-c at the GDB window
occasionally and making a backtrace.  Then continue and stop it again,
and make another backtrace, etc.  After several backtraces you may
start to see a pattern that shows where it is spending the time.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

Richard M. Stallman wrote:
> The way to investigate this is to run Emacs with X, under GDB, and try
> stopping it during this operation by typing C-c at the GDB window
> occasionally and making a backtrace.  Then continue and stop it again,
> and make another backtrace, etc.  After several backtraces you may
> start to see a pattern that shows where it is spending the time.

OK, I started a new emacs a couple of days ago under gdb.  I ran the
lisp snippet scanning /usr/bin and it reported 0 seconds.  One day
later it gave 3 seconds (so already some slowdown).  I repeated it and
did some gdb backtraces while it was running.  Today the lisp snippet
took 7 seconds to run.  Again, I repeated the call and took several
backtraces.

The calls seem to be in a fairly even balance of mixture of
Fget_buffer and Fgarbage_collect.  Why creating a list of files in a
directory should be calling get_buffer is beyond me. list-buffers
shows about 40 buffers, and when I try switch-to-buffer with a leading
space, there are about 200 buffers listed (about 170 of which are
named like *code-conversion-work*<nnn>)

Cheers,
Len.

--- Day One Backtraces ---
(gdb) where
#0  Fget_buffer (name=196878843) at buffer.c:261
#1  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#2  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#3  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=195343227, from=0, from_byte=0, to=6, to_byte=6,
    dst_object=137820409) at coding.c:7201
#4  0x080b886e in code_convert_string (string=195343227, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#5  0x080b89e8 in code_convert_string_norecord (string=195343227, coding_system=138040193, encodep=0) at coding.c:8525
#6  0x08148e6d in directory_files_internal (directory=196579291, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#7  0x0814917a in Fdirectory_files (directory=196579291, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#8  0x0817b194 in Feval (form=189817397) at eval.c:2383

(gdb) where
#0  0x081845bc in Fstring_equal (s1=184620323, s2=193968227) at fns.c:233
#1  0x08133cbd in Fget_buffer (name=193968227) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=193516003, from=0, from_byte=0, to=12, to_byte=12,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=193516003, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=193516003, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=196579291, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=196579291, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=189817397) at eval.c:2383

(gdb) where
[many pages omitted]
#12423 0x0816393e in mark_object (arg=137972241) at alloc.c:5545
#12424 0x08163a1f in mark_object (arg=179302621) at alloc.c:5655
#12425 0x0816393e in mark_object (arg=138015521) at alloc.c:5545
#12426 0x08163a1f in mark_object (arg=186491733) at alloc.c:5655
#12427 0x0816393e in mark_object (arg=137820361) at alloc.c:5545
#12428 0x0816393e in mark_object (arg=137820385) at alloc.c:5545
#12429 0x08163928 in mark_object (arg=191577785) at alloc.c:5543
#12430 0x08163def in mark_vectorlike (ptr=0x836fce0) at alloc.c:5365
#12431 0x08166dff in Fgarbage_collect () at alloc.c:5078
#12432 0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#12433 0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#12434 0x081355d5 in Fkill_buffer (buffer=197675196) at buffer.c:1445
#12435 0x080b901f in code_conversion_restore (arg=188523421) at coding.c:7005
#12436 0x08179b1d in unbind_to (count=160, value=197674755) at eval.c:3397
#12437 0x080b886e in code_convert_string (string=197725859, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#12438 0x080b89e8 in code_convert_string_norecord (string=197725859, coding_system=138040193, encodep=0) at coding.c:8525
#12439 0x08148e6d in directory_files_internal (directory=196579291, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#12440 0x0814917a in Fdirectory_files (directory=196579291, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#12441 0x0817b194 in Feval (form=189817397) at eval.c:2383

(gdb) where
[many pages omitted]
#12143 0x08163933 in mark_object (arg=147731373) at alloc.c:5544
#12144 0x0816393e in mark_object (arg=137972241) at alloc.c:5545
#12145 0x08163a1f in mark_object (arg=179302621) at alloc.c:5655
#12146 0x0816393e in mark_object (arg=138015521) at alloc.c:5545
#12147 0x08163a1f in mark_object (arg=186491733) at alloc.c:5655
#12148 0x0816393e in mark_object (arg=137820361) at alloc.c:5545
#12149 0x0816393e in mark_object (arg=137820385) at alloc.c:5545
#12150 0x08163928 in mark_object (arg=191577785) at alloc.c:5543
#12151 0x08163def in mark_vectorlike (ptr=0x836fce0) at alloc.c:5365
#12152 0x08166dff in Fgarbage_collect () at alloc.c:5078
#12153 0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#12154 0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#12155 0x081355d5 in Fkill_buffer (buffer=195696876) at buffer.c:1445
#12156 0x080b901f in code_conversion_restore (arg=189705981) at coding.c:7005
#12157 0x08179b1d in unbind_to (count=160, value=197035867) at eval.c:3397
#12158 0x080b886e in code_convert_string (string=197033387, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#12159 0x080b89e8 in code_convert_string_norecord (string=197033387, coding_system=138040193, encodep=0) at coding.c:8525
#12160 0x08148e6d in directory_files_internal (directory=195167243, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#12161 0x0814917a in Fdirectory_files (directory=195167243, full=137820361, match=137820361, nosort=137820361) at dired.c:347

(gdb) where
#0  Fgarbage_collect () at alloc.c:5856
#1  0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#2  0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#3  0x081355d5 in Fkill_buffer (buffer=197100188) at buffer.c:1445
#4  0x080b901f in code_conversion_restore (arg=188059837) at coding.c:7005
#5  0x08179b1d in unbind_to (count=160, value=197100003) at eval.c:3397
#6  0x080b886e in code_convert_string (string=197088803, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#7  0x080b89e8 in code_convert_string_norecord (string=197088803, coding_system=138040193, encodep=0) at coding.c:8525
#8  0x08148e6d in directory_files_internal (directory=195167243, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#9  0x0814917a in Fdirectory_files (directory=195167243, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#10 0x0817b194 in Feval (form=194228701) at eval.c:2383



--- Day Two Backtraces ---
(gdb) where -50
#0  0x08184629 in Fstring_equal (s1=196065283, s2=208078795) at fns.c:233
#1  0x08133cbd in Fget_buffer (name=208078795) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=208096283, from=0, from_byte=0, to=10, to_byte=10,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=208096283, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=208096283, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=216322659, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=216322659, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=193837069) at eval.c:2383
#

(gdb) where -50
#0  0x081845bc in Fstring_equal (s1=190322379, s2=221887595) at fns.c:233
#1  0x08133cbd in Fget_buffer (name=221887595) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=202026699, from=0, from_byte=0, to=7, to_byte=7,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=202026699, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=202026699, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=216322659, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=216322659, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#

(gdb) where -50
#0  0x08163918 in mark_object (arg=137820361) at alloc.c:5540
#1  0x08163def in mark_vectorlike (ptr=0xb064218) at alloc.c:5365
#2  0x08163def in mark_vectorlike (ptr=0xb046658) at alloc.c:5365
#3  0x08163def in mark_vectorlike (ptr=0xb041f90) at alloc.c:5365
#4  0x08163def in mark_vectorlike (ptr=0xb041e70) at alloc.c:5365
#5  0x08163def in mark_vectorlike (ptr=0x8408178) at alloc.c:5365
#6  0x08163def in mark_vectorlike (ptr=0x83def70) at alloc.c:5365
#7  0x08166dff in Fgarbage_collect () at alloc.c:5078
#8  0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#9  0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#10 0x081355d5 in Fkill_buffer (buffer=222457108) at buffer.c:1445
#11 0x080b901f in code_conversion_restore (arg=194838373) at coding.c:7005
#12 0x08179b1d in unbind_to (count=160, value=198800851) at eval.c:3397
#13 0x080b886e in code_convert_string (string=198853539, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#14 0x080b89e8 in code_convert_string_norecord (string=198853539, coding_system=138040193, encodep=0) at coding.c:8525
#15 0x08148e6d in directory_files_internal (directory=216322659, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#16 0x0814917a in Fdirectory_files (directory=216322659, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#17 0x0817b194 in Feval (form=193837069) at eval.c:2383
#

(gdb) where -50
#0  0x08184629 in Fstring_equal (s1=189869411, s2=196249035) at fns.c:233
#1  0x08133cbd in Fget_buffer (name=196249035) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=220991979, from=0, from_byte=0, to=8, to_byte=8,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=220991979, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=220991979, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=216322659, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=216322659, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=193837069) at eval.c:2383
#

(gdb) where -50
#0  0x081845be in Fstring_equal (s1=190912227, s2=217450235) at fns.c:233
#1  0x08133cbd in Fget_buffer (name=217450235) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=217882579, from=0, from_byte=0, to=10, to_byte=10,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=217882579, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=217882579, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=214603771, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=214603771, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=191831085) at eval.c:2383
#

(gdb) where -50
#0  0x081845bc in Fstring_equal (s1=191578811, s2=195676059) at fns.c:233
#1  0x08133cbd in Fget_buffer (name=195676059) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=195551091, from=0, from_byte=0, to=17, to_byte=17,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=195551091, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=195551091, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=214603771, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=214603771, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=191831085) at eval.c:2383
#

(gdb) where -50
#0  0x0816390f in mark_object (arg=142625740) at alloc.c:5678
#1  0x08163def in mark_vectorlike (ptr=0x8800410) at alloc.c:5365
#2  0x08163def in mark_vectorlike (ptr=0x8760d60) at alloc.c:5365
#3  0x08163def in mark_vectorlike (ptr=0x838a740) at alloc.c:5365
#4  0x08166dff in Fgarbage_collect () at alloc.c:5078
#5  0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#6  0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#7  0x081355d5 in Fkill_buffer (buffer=222055956) at buffer.c:1445
#8  0x080b901f in code_conversion_restore (arg=192601293) at coding.c:7005
#9  0x08179b1d in unbind_to (count=160, value=222055579) at eval.c:3397
#10 0x080b886e in code_convert_string (string=222243771, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#11 0x080b89e8 in code_convert_string_norecord (string=222243771, coding_system=138040193, encodep=0) at coding.c:8525
#12 0x08148e6d in directory_files_internal (directory=213694619, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#13 0x0814917a in Fdirectory_files (directory=213694619, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#14 0x0817b194 in Feval (form=194208933) at eval.c:2383
#


#603 0x0816393e in mark_object (arg=137820361) at alloc.c:5545
#604 0x0816393e in mark_object (arg=137820385) at alloc.c:5545
#605 0x08163928 in mark_object (arg=191577785) at alloc.c:5543
#606 0x08163def in mark_vectorlike (ptr=0x836fce0) at alloc.c:5365
#607 0x08166dff in Fgarbage_collect () at alloc.c:5078
#608 0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#609 0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#610 0x081355d5 in Fkill_buffer (buffer=194289132) at buffer.c:1445
#611 0x080b901f in code_conversion_restore (arg=194202789) at coding.c:7005
#612 0x08179b1d in unbind_to (count=160, value=198722819) at eval.c:3397
#613 0x080b886e in code_convert_string (string=221890867, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#614 0x080b89e8 in code_convert_string_norecord (string=221890867, coding_system=138040193, encodep=0) at coding.c:8525
#615 0x08148e6d in directory_files_internal (directory=213694619, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#616 0x0814917a in Fdirectory_files (directory=213694619, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#617 0x0817b194 in Feval (form=194208933) at eval.c:2383
#

(gdb) where -50
#0  0x0816719b in Fgarbage_collect () at alloc.c:2218
#1  0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#2  0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#3  0x081355d5 in Fkill_buffer (buffer=212512388) at buffer.c:1445
#4  0x080b901f in code_conversion_restore (arg=194835013) at coding.c:7005
#5  0x08179b1d in unbind_to (count=160, value=196608707) at eval.c:3397
#6  0x080b886e in code_convert_string (string=198721867, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#7  0x080b89e8 in code_convert_string_norecord (string=198721867, coding_system=138040193, encodep=0) at coding.c:8525
#8  0x08148e6d in directory_files_internal (directory=213694619, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#9  0x0814917a in Fdirectory_files (directory=213694619, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#10 0x0817b194 in Feval (form=194208933) at eval.c:2383
#

#770 0x08163928 in mark_object (arg=191577785) at alloc.c:5543
#771 0x08163def in mark_vectorlike (ptr=0x836fce0) at alloc.c:5365
#772 0x08166dff in Fgarbage_collect () at alloc.c:5078
#773 0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#774 0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#775 0x081355d5 in Fkill_buffer (buffer=198416924) at buffer.c:1445
#776 0x080b901f in code_conversion_restore (arg=192601365) at coding.c:7005
#777 0x08179b1d in unbind_to (count=160, value=198227619) at eval.c:3397
#778 0x080b886e in code_convert_string (string=196609195, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#779 0x080b89e8 in code_convert_string_norecord (string=196609195, coding_system=138040193, encodep=0) at coding.c:8525
#780 0x08148e6d in directory_files_internal (directory=216744507, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#781 0x0814917a in Fdirectory_files (directory=216744507, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#782 0x0817b194 in Feval (form=209294021) at eval.c:2383
#

#598 0x0816393e in mark_object (arg=137820361) at alloc.c:5545
#599 0x0816393e in mark_object (arg=137820385) at alloc.c:5545
#600 0x08163928 in mark_object (arg=191577785) at alloc.c:5543
#601 0x08163def in mark_vectorlike (ptr=0x836fce0) at alloc.c:5365
#602 0x08166dff in Fgarbage_collect () at alloc.c:5078
#603 0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#604 0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#605 0x081355d5 in Fkill_buffer (buffer=198216204) at buffer.c:1445
#606 0x080b901f in code_conversion_restore (arg=194202981) at coding.c:7005
#607 0x08179b1d in unbind_to (count=160, value=222454971) at eval.c:3397
#608 0x080b886e in code_convert_string (string=198213795, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#609 0x080b89e8 in code_convert_string_norecord (string=198213795, coding_system=138040193, encodep=0) at coding.c:8525
#610 0x08148e6d in directory_files_internal (directory=216744507, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#611 0x0814917a in Fdirectory_files (directory=216744507, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#612 0x0817b194 in Feval (form=209294021) at eval.c:2383
#

#601 0x0816393e in mark_object (arg=137820385) at alloc.c:5545
#602 0x08163928 in mark_object (arg=191577785) at alloc.c:5543
#603 0x08163def in mark_vectorlike (ptr=0x836fce0) at alloc.c:5365
#604 0x08166dff in Fgarbage_collect () at alloc.c:5078
#605 0x0817b9cc in Ffuncall (nargs=1, args=0xbfaa3d58) at eval.c:2976
#606 0x0817d0d1 in run_hook_with_args (nargs=1, args=0xbfaa3d58, cond=until_failure) at eval.c:2701
#607 0x081355d5 in Fkill_buffer (buffer=202062812) at buffer.c:1445
#608 0x080b901f in code_conversion_restore (arg=194834997) at coding.c:7005
#609 0x08179b1d in unbind_to (count=160, value=221136507) at eval.c:3397
#610 0x080b886e in code_convert_string (string=222456051, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#611 0x080b89e8 in code_convert_string_norecord (string=222456051, coding_system=138040193, encodep=0) at coding.c:8525
#612 0x08148e6d in directory_files_internal (directory=216744507, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#613 0x0814917a in Fdirectory_files (directory=216744507, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#614 0x0817b194 in Feval (form=209294021) at eval.c:2383
#

(gdb) where -50
#0  0x0818459c in Fstring_equal (s1=194183139, s2=211406875) at fns.c:230
#1  0x08133cbd in Fget_buffer (name=211406875) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=211990507, from=0, from_byte=0, to=11, to_byte=11,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=211990507, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=211990507, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=216744507, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=216744507, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=209294021) at eval.c:2383
#

(gdb) where -50
#0  Fstring_equal (s1=192511995, s2=195140291) at fns.c:236
#1  0x08133cbd in Fget_buffer (name=195140291) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=195140227, from=0, from_byte=0, to=9, to_byte=9,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=195140227, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=195140227, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=217102067, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=217102067, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=196791149) at eval.c:2383
#

(gdb) where -50
#0  Fstring_equal (s1=192565003, s2=190438459) at fns.c:230
#1  0x08133cbd in Fget_buffer (name=190438459) at buffer.c:261
#2  0x08133fc7 in Fgenerate_new_buffer_name (name=137883971, ignore=137820361) at buffer.c:867
#3  0x080b78ac in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6970
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=190433283, from=0, from_byte=0, to=26, to_byte=26,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=190433283, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=190433283, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=217102067, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=217102067, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=196791149) at eval.c:2383
#

(gdb) where -50
#0  0x08184629 in Fstring_equal (s1=190379323, s2=200546291) at fns.c:233
#1  0x08133cbd in Fget_buffer (name=200546291) at buffer.c:261
#2  0x08134644 in Fget_buffer_create (name=200546291) at buffer.c:347
#3  0x080b78b4 in code_conversion_save (with_work_buf=1, multibyte=1) at coding.c:6971
#4  0x080b83da in decode_coding_object (coding=0xbfaa3dec, src_object=202101795, from=0, from_byte=0, to=23, to_byte=23,
    dst_object=137820409) at coding.c:7201
#5  0x080b886e in code_convert_string (string=202101795, coding_system=138040193, dst_object=137820409, encodep=0, nocopy=0,
    norecord=1) at coding.c:8504
#6  0x080b89e8 in code_convert_string_norecord (string=202101795, coding_system=138040193, encodep=0) at coding.c:8525
#7  0x08148e6d in directory_files_internal (directory=217102067, full=137820361, match=137820361, nosort=137820361, attrs=0,
    id_format=137820361) at dired.c:237
#8  0x0814917a in Fdirectory_files (directory=217102067, full=137820361, match=137820361, nosort=137820361) at dired.c:347
#9  0x0817b194 in Feval (form=196791149) at eval.c:2383
#

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Chong Yidong]

Len Trigg <lenbok@gmail.com> writes:

> OK, I started a new emacs a couple of days ago under gdb.  I ran the
> lisp snippet scanning /usr/bin and it reported 0 seconds... Today the
> lisp snippet took 7 seconds to run.  Again, I repeated the call and
> took several backtraces.
>
> The calls seem to be in a fairly even balance of mixture of
> Fget_buffer and Fgarbage_collect.  Why creating a list of files in a
> directory should be calling get_buffer is beyond me. list-buffers
> shows about 40 buffers, and when I try switch-to-buffer with a leading
> space, there are about 200 buffers listed (about 170 of which are
> named like *code-conversion-work*<nnn>)

That's very odd: those buffers are used for code conversion, and they
ought to be killed automatically.

Could you check if you have anything unusual in
kill-buffer-query-functions or kill-buffer hook?

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

At Thu, 11 Sep 2008 13:08:07 -0400,
Richard M. Stallman wrote:
> 
>     The calls seem to be in a fairly even balance of mixture of
>     Fget_buffer and Fgarbage_collect.  Why creating a list of files in a
>     directory should be calling get_buffer is beyond me.
> 
> The backtrace shows why.  It is making a new code-conversion buffer.

Yes, but as an emacs user (rather than programmer), I think of buffers
as being fairly heavyweight things (after all, they contain all the
text I'm editing, and a typical user probably only has a few dozen
open), compared to calling some kind of straight string conversion
function.  Perhaps that is an invalid assumption and buffers are
lightweight.

> There must be a bug in the logic that decides whether to make a new
> code-conversion buffer, so that it makes too many of them.

It seems this is likely to be the problem.  This morning there are 225
code-conversion-work buffers hanging around, and my directory listing
takes 11 seconds.


Cheers,
Len.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Chong Yidong]

> It seems this is likely to be the problem.  This morning there are 225
> code-conversion-work buffers hanging around, and my directory listing
> takes 11 seconds.

What is the value of kill-buffer-query-functions and kill-buffer-hook?

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

At Thu, 11 Sep 2008 20:42:38 -0400,
Chong Yidong wrote:
> 
> > It seems this is likely to be the problem.  This morning there are 225
> > code-conversion-work buffers hanging around, and my directory listing
> > takes 11 seconds.
> 
> What is the value of kill-buffer-query-functions and kill-buffer-hook?

(server-kill-buffer-query-function)

and

(tramp-flush-file-function erc-kill-buffer-function browse-url-delete-temp-file jde-db-nullify-breakpoint-markers vc-kill-buffer-hook)

I can try restarting my emacs without using some of those features for
the next few days.

Cheers,
Len.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

Len Trigg wrote:
> (tramp-flush-file-function erc-kill-buffer-function browse-url-delete-temp-file jde-db-nullify-breakpoint-markers vc-kill-buffer-hook)
> 
> I can try restarting my emacs without using some of those features for
> the next few days.

I think I have some more information on where those extra
code-conversion-work buffers are coming from.  I have a crontab
scheduled script that regularly downloads ics calendar files from
various locations and uses icalendar.el to convert them to diary
format (which are then #included by my main diary file).  This
conversion uses a separate "emacs -batch" process in order to not
interfere with my main emacs, and then finally calls emacsclient at
the end to tell my main emacs to update it's current diary display, if
present.

If I disable this script entirely, the code-conversion-work buffers
don't accumulate.

If I disable the final call to emacsclient, the code-conversion-work
buffers do still accumulate (eliminating emacsclient as the culprit).

If I disable global-auto-revert mode in my main emacs, the
code-conversion-work buffers don't accumulate.

So it seems to be related to the modified diary files being
auto-reverted.  I have not managed to work out any more details than
that.


Regardless of why they are leaking, it seems to me that Richards
suggestion of eliminating the use of multiple code-conversion buffers
at all is probably the way to go.


Cheers,
Len.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Kenichi Handa]

In article <E1KdpeJ-0004Sb-Oj@fencepost.gnu.org>, "Richard M. Stallman" <rms@gnu.org> writes:

> There must be a bug in the logic that decides whether to make a new
> code-conversion buffer, so that it makes too many of them.

> The code looks inefficient too.  Even when it reuses the
> code-conversion buffer, it calls Fget_buffer_create.
> It seems to be killing and recreating these buffers,
> which must be slow.

???  I don't see what's wrong with the current code.  I
think the functions make_conversion_work_buffer,
code_conversion_save, and code_conversion_restore do the
right thing.

> I think it would be better to make one and never kill it.

> Handa-san, under what circumstances should it need to use more than
> one of these buffers?  Can character set encoding and decoding ever be
> done recursively?  It seems to me that that should never happen, so
> one such buffer should be enough.

There are two cases that multiple work buffers are necessary.

(1) A coding system can have pre-write-conversion function,
and that function can call code-conversion function
recursively.

(2) If REPLACE arg is non-nil in insert-file-contents and a
file need a decoding, we call decode_coding_c_string while a
work buffer is in use.

Anyway, I fixed one bug in insert-file-contents.  It failed
to run code_conversion_restore in one situation.  So, could
you try again with the latest code?

---
Kenichi Handa
handa@ni.aist.go.jp

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Len Trigg]

Kenichi Handa wrote:
> Anyway, I fixed one bug in insert-file-contents.  It failed
> to run code_conversion_restore in one situation.  So, could
> you try again with the latest code?

I have updated to current CVS and tried running my script a few times.
So far no extra code_conversion_work buffers have appeared, so this
appears to be fixed!  My emacs may live long and prosper (rather than
having to be restarted every few days :-))


Cheers,
Len.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Richard M. Stallman]

    (1) A coding system can have pre-write-conversion function,
    and that function can call code-conversion function
    recursively.

I read in the doc string of define-coding-system
that the pre-write-conversion function is called
before doing the ordinary work of encoding.

Is it possible to call the pre-write-conversion function
before obtaining the work buffer?  That way, there would never
be recursive encoding.

    (2) If REPLACE arg is non-nil in insert-file-contents and a
    file need a decoding, we call decode_coding_c_string while a
    work buffer is in use.

I see why decoding is needed in this case, but why is a work
buffer already in use?  It seems to me that there is only one
act of decoding to be done in that call to insert-file-contents.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Richard M. Stallman]

    > The code looks inefficient too.  Even when it reuses the
    > code-conversion buffer, it calls Fget_buffer_create.
    > It seems to be killing and recreating these buffers,
    > which must be slow.

    ???  I don't see what's wrong with the current code.  I
    think the functions make_conversion_work_buffer,
    code_conversion_save, and code_conversion_restore do the
    right thing.

They do the right thing, but it looks like they do it inefficiently.

Whether that is a significant slowdown, I do not know.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Kenichi Handa]

In article <E1KfIlu-00027F-0p@fencepost.gnu.org>, "Richard M. Stallman" <rms@gnu.org> writes:

>     (1) A coding system can have pre-write-conversion function,
>     and that function can call code-conversion function
>     recursively.

> I read in the doc string of define-coding-system
> that the pre-write-conversion function is called
> before doing the ordinary work of encoding.

> Is it possible to call the pre-write-conversion function
> before obtaining the work buffer?

No.  If the source of encoding is a string, it is inserted
in a working buffer to allow pre-write-conversion function
to modify it.

>     (2) If REPLACE arg is non-nil in insert-file-contents and a
>     file need a decoding, we call decode_coding_c_string while a
>     work buffer is in use.

> I see why decoding is needed in this case, but why is a work
> buffer already in use?  It seems to me that there is only one
> act of decoding to be done in that call to insert-file-contents.

insert-file-contents uses the working buffer to hold the
output of decoding.  So, when decode_coding_c_string is
called, we have already called code_conversion_save to make
the working buffer.

---
Kenichi Handa
handa@ni.aist.go.jp

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Kenichi Handa]

In article <E1KfImG-0002AY-GR@fencepost.gnu.org>, "Richard M. Stallman" <rms@gnu.org> writes:

> The code looks inefficient too.  Even when it reuses the
> code-conversion buffer, it calls Fget_buffer_create.
> It seems to be killing and recreating these buffers,
> which must be slow.

>     ???  I don't see what's wrong with the current code.  I
>     think the functions make_conversion_work_buffer,
>     code_conversion_save, and code_conversion_restore do the
>     right thing.

> They do the right thing, but it looks like they do it inefficiently.

I've just modified the current code to avoid calling
Fget_buffer_create when we alreay have
Vcode_conversion_reused_workbuf.  But, I think it doens't
influence the efficiency that much because
Fget_buffer_create doesn't create a new buffer if a buffer
of the specified name already exists.

I think the current problem of slowness was fixed by a fix
of Finsert_file_contents as in my previous mail.

---
Kenichi Handa
handa@ni.aist.go.jp

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Stefan Monnier]

> I've just modified the current code to avoid calling
> Fget_buffer_create when we alreay have
> Vcode_conversion_reused_workbuf.  But, I think it doens't
> influence the efficiency that much because
> Fget_buffer_create doesn't create a new buffer if a buffer
> of the specified name already exists.

But the title-problem showed that having many buffers (like 200 or so)
can cause this buffer-name-lookup to take a long time.  The worst part
is that having 200 buffers is not particularly excessive: I regularly
have Emacs session with a hundred buffers or so.  So we should avoid
this lookup loop whenever necessary.


        Stefan

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Richard M. Stallman]

    No.  If the source of encoding is a string, it is inserted
    in a working buffer to allow pre-write-conversion function
    to modify it.

I see.

Thanks.

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Kenichi Handa]

In article <jwvwshc9qfw.fsf-monnier+emacsbugreports@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > I've just modified the current code to avoid calling
> > Fget_buffer_create when we alreay have
> > Vcode_conversion_reused_workbuf.  But, I think it doens't
> > influence the efficiency that much because
> > Fget_buffer_create doesn't create a new buffer if a buffer
> > of the specified name already exists.

> But the title-problem showed that having many buffers (like 200 or so)
> can cause this buffer-name-lookup to take a long time.  The worst part
> is that having 200 buffers is not particularly excessive: I regularly
> have Emacs session with a hundred buffers or so.  So we should avoid
> this lookup loop whenever necessary.

The slowness came from the inefficiency of
generate-new-buffer-name when there exist many buffers of
the same base name.  To generate the 201th buffer, it scans
the buffer list 200 times (comparing names 200*201/2 = 20100
times), and the following get-buffer-create scans it again.

As there usually exist a few buffers whose name start with a
space, I believe get-buffer is very fast for finding such a
buffer.  But, even if that is so, in genenral, I agree that
we should avoid unnecessary call.

---
Kenichi Handa
handa@ni.aist.go.jp

bug#122: 23.0.60; Slowdown in directory scanning over time.

[Stefan Monnier]

> The slowness came from the inefficiency of
> generate-new-buffer-name when there exist many buffers of
> the same base name.  To generate the 201th buffer, it scans
> the buffer list 200 times (comparing names 200*201/2 = 20100
> times), and the following get-buffer-create scans it again.

Oh, yes, of course.  When generate-new-buffer-name is called for
a user-visible buffer, fixing this would be probably too much trouble
for too little benefit.  But for internal buffers, whose precise name
doesn't actually matter, we should use a different strategy, where we
immediately start by adding a random suffix to the buffer name, so as to
avoid conflicts.


        Stefan