emacs slow on dired renaming

emacs slow on dired renaming

[Robert Marshall]

I have a build of emacs from the bzr trunk (of last week) for which
dired renaming of files is very slow. It takes around 60 seconds to move
16 files to a subdirectory of their current location (within the same
filesystem). The delay is spread across the files so each rename appears
to take the same (c 5 seconds) time.

I've not yet managed to construct a minimal test case for this - if I
start -Q or with my .emacs - without loading desktop the same rename
runs virtually instantaneously.

I am seeing a similar slowness in VM (the mail reader) which I suspect is
related - here the problem gets worse the longer the emacs session is
left running - so if it is the same problem as above maybe that's why a
simple with -Q doesn't show the problem?

I'm not placing it is a bug report yet as this is rather vague but in
case anyone has any helpful thoughts or is also seeing this issue?

In GNU Emacs 24.4.50.2 (i686-pc-linux-gnu, GTK+ Version 3.8.6)
 of 2014-03-27 on poulenc
Repository revision: 116883 lekktu@gmail.com-20140327011754-63o9myvyaubkzn1b
Windowing system distributor `The X.Org Foundation', version 11.0.11405000
System Description:	Ubuntu 13.10

Robert
-- 
La grenouille songe..dans son château d'eau
Links and things http://rmstar.blogspot.com/

Re: emacs slow on dired renaming

[Eli Zaretskii]

> From: Robert Marshall <spam@capuchin.co.uk>
> Date: Thu, 03 Apr 2014 07:57:14 +0100
> 
> I have a build of emacs from the bzr trunk (of last week) for which
> dired renaming of files is very slow. It takes around 60 seconds to move
> 16 files to a subdirectory of their current location (within the same
> filesystem). The delay is spread across the files so each rename appears
> to take the same (c 5 seconds) time.

Try profiling Emacs with "M-x profile-start".  That might show you a
hint where the delay hides.

Re: emacs slow on dired renaming

[Robert Marshall]

On Thu, Apr 03 2014, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Robert Marshall <spam@capuchin.co.uk>
>> Date: Thu, 03 Apr 2014 07:57:14 +0100
>> 
>> I have a build of emacs from the bzr trunk (of last week) for which
>> dired renaming of files is very slow. It takes around 60 seconds to move
>> 16 files to a subdirectory of their current location (within the same
>> filesystem). The delay is spread across the files so each rename appears
>> to take the same (c 5 seconds) time.
>
> Try profiling Emacs with "M-x profile-start".  That might show you a
> hint where the delay hides.
>

It appears that the delay gradually increases when it does go bad. At
the moment a rename is taking around 1 sec (same copy operation is virtually
instantaneous when emacs first started)

This (cpu report) was with moving 10 files
- command-execute                                                4937  99%
 - call-interactively                                            4937  99%
  - dired-do-rename                                              4844  97%
   - dired-do-create-files                                       4844  97%
    - dired-create-files                                         4763  96%
     - byte-code                                                 4728  95%
      - dired-rename-file                                        3510  70%
       - dired-rename-subdir                                     2276  45%
        - dired-fun-in-all-buffers                               1187  23%
         - dired-buffers-for-dir                                 1183  23%
            dired-in-this-tree                                     13   0%
       - dired-remove-file                                       1222  24%
        - dired-fun-in-all-buffers                               1222  24%
         - dired-buffers-for-dir                                 1186  23%
            dired-in-this-tree                                      4   0%
         + apply                                                   28   0%
      - dired-add-file                                           1214  24%
       - dired-fun-in-all-buffers                                1214  24%
          dired-buffers-for-dir                                  1184  23%
        + apply                                                    30   0%
     + dired-file-marker                                           35   0%
    + dired-mark-read-file-name                                    58   1%
      dired-get-marked-files                                       19   0%
    + mapcar                                                        4   0%
  + byte-code                                                      62   1%
  + minibuffer-complete                                            21   0%
  + execute-extended-command                                       10   0%
+ timer-event-handler                                               8   0%
+ redisplay_internal (C function)                                   4   0%
+ ...                                                               0   0%

so the delay appears spread across various routines? I did have a report with
the very long delay and I think it was similar[1] - certainly there wasn't
one obvious routine which was hogging the CPU. The system monitor shows
emacs at 100% of 1 cpu while it does the rename.

Robert
[1] unfortunately I forgot to save it when restarting to get a fresh
emacs :-(
-- 
La grenouille songe..dans son château d'eau
Links and things http://rmstar.blogspot.com/

Re: emacs slow on dired renaming

[Eli Zaretskii]

> From: Robert Marshall <spam@capuchin.co.uk>
> Date: Thu, 03 Apr 2014 19:57:44 +0100
> 
> > Try profiling Emacs with "M-x profile-start".  That might show you a
> > hint where the delay hides.
> >
> 
> It appears that the delay gradually increases when it does go bad. At
> the moment a rename is taking around 1 sec (same copy operation is virtually
> instantaneous when emacs first started)

Maybe wait until it gets slower and profile again.

Also, manually load dired.el and dired-aux.el (not the *.elc files),
so that more Lisp functions show instead of the unhelpful "byte-code".

Re: emacs slow on dired renaming

[Robert Marshall]

On Thu, Apr 03 2014, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Robert Marshall <spam@capuchin.co.uk>
>> Date: Thu, 03 Apr 2014 19:57:44 +0100
>> 
>> > Try profiling Emacs with "M-x profile-start".  That might show you a
>> > hint where the delay hides.
>> >
>> 
>> It appears that the delay gradually increases when it does go bad. At
>> the moment a rename is taking around 1 sec (same copy operation is virtually
>> instantaneous when emacs first started)
>
> Maybe wait until it gets slower and profile again.
>
> Also, manually load dired.el and dired-aux.el (not the *.elc files),
> so that more Lisp functions show instead of the unhelpful "byte-code".
>

At the moment it's still under 2 sec for each file. I've manually loaded
dired and dired-aux and now get the report which follows. The culprit
appears to be dired-in-this-tree (or the number of times its called as
its rather short) - at the moment I have 42 dired buffers open (and
dired-buffers confirms this) of which only 1 or 2 have subdirectories
also open within the buffer.

I've opened a couple more dired buffers since starting but most of them were
open from the beginning (via desktop) and then there was no slowdown.

     - dired-create-files                                      26704  94%
       - let                                                    26704  94%
        - let                                                   26701  94%
         - let                                                  26701  94%
          - while                                               26701  94%
           - if                                                 26701  94%
            - let*                                              26701  94%
             - condition-case                                   26448  93%
              - progn                                           26448  93%
               - funcall                                        21434  75%
                - dired-rename-file                             21434  75%
                 - dired-rename-subdir                          15928  56%
                  - let                                         10940  38%
                   - while                                      10936  38%
                    - save-current-buffer                       10878  38%
                     - if                                       10855  38%
                      - and                                     10855  38%
                       - dired-in-this-tree                     10841  38%
                        + let                                      23   0%
                      setq                                          4   0%
                  + dired-fun-in-all-buffers                     4988  17%
                 - dired-remove-file                             5505  19%
                  - dired-fun-in-all-buffers                     5505  19%
                   - let                                         5505  19%
                    - let                                        5505  19%
                     - dired-buffers-for-dir                     5118  18%
                      - let                                      5118  18%
                       - let                                     5118  18%
                        - while                                  5118  18%
                         - let                                   5118  18%
                          - cond                                 5110  17%
                           - dired-in-this-tree                  5110  17%
                            + let                                  20   0%
                     + while                                      387   1%
               - dired-add-file                                  5014  17%
                - dired-fun-in-all-buffers                       5014  17%
                 - let                                           5014  17%
                  - let                                          5014  17%
                   - dired-buffers-for-dir                       5014  17%
                    - let                                        5014  17%
                     - let                                       5014  17%
                      - while                                    5014  17%
                       - let                                     5010  17%
                        - cond                                   5004  17%
                         - dired-in-this-tree                    5004  17%
                          - let                                    14   0%
 
-- 
La grenouille songe..dans son château d'eau
Links and things http://rmstar.blogspot.com/

Re: emacs slow on dired renaming

[Eli Zaretskii]

> From: Robert Marshall <spam@capuchin.co.uk>
> Date: Fri, 04 Apr 2014 14:17:51 +0100
> 
> At the moment it's still under 2 sec for each file.

When you start a new session, does the command run as fast as in
"emacs -Q", or is it already significantly slower?  If the latter, it
might be better to bisect your .emacs until you find the part that
causes the slowdown.

> I've manually loaded
> dired and dired-aux and now get the report which follows. The culprit
> appears to be dired-in-this-tree (or the number of times its called as
> its rather short) - at the moment I have 42 dired buffers open (and
> dired-buffers confirms this) of which only 1 or 2 have subdirectories
> also open within the buffer.
> 
> I've opened a couple more dired buffers since starting but most of them were
> open from the beginning (via desktop) and then there was no slowdown.

I see nothing of interest in the profile.

Does this happen with renaming any file/directory, or only with files
on some specific filesystems?  Are those filesystems local ore remote
(NFS, Samba, etc.)?

If you try renaming a single file with "M-x rename-file RET", does it
also take 2 sec?

Re: emacs slow on dired renaming

[Robert Marshall]

On Fri, Apr 04 2014, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Robert Marshall <spam@capuchin.co.uk>
>> Date: Fri, 04 Apr 2014 14:17:51 +0100
>> 
>> At the moment it's still under 2 sec for each file.
>
> When you start a new session, does the command run as fast as in
> "emacs -Q", or is it already significantly slower?  If the latter, it
> might be better to bisect your .emacs until you find the part that
> causes the slowdown.
>

In both cases (with and without -Q) renaming the 10 files - moving them
from one directory to a subdirectory - (with both dired buffers
displayed in the frame and using dired-dwim-target set to t to provide
the default target and that's the same as in the slow case) there is no
perceptible difference - both appear to be immediate.

>> I've manually loaded
>> dired and dired-aux and now get the report which follows. The culprit
>> appears to be dired-in-this-tree (or the number of times its called as
>> its rather short) - at the moment I have 42 dired buffers open (and
>> dired-buffers confirms this) of which only 1 or 2 have subdirectories
>> also open within the buffer.
>> 
>> I've opened a couple more dired buffers since starting but most of them were
>> open from the beginning (via desktop) and then there was no slowdown.
>
> I see nothing of interest in the profile.
>
> Does this happen with renaming any file/directory, or only with files
> on some specific filesystems?  Are those filesystems local ore remote
> (NFS, Samba, etc.)?
>

They're both directories within $HOME which is /dev/sdb7 - so both local.
I'd suspect a dodgy disk if it wasn't for the delay vanishing on a
restart - and I've been trying a new emacs session at the same time as
the one which produces the delay - and the new one has no delay.

> If you try renaming a single file with "M-x rename-file RET", does it
> also take 2 sec?
>
There's no apparent delay

Robert
-- 
La grenouille songe..dans son château d'eau
Links and things http://rmstar.blogspot.com/

Re: emacs slow on dired renaming

[Eli Zaretskii]

> From: Robert Marshall <spam@capuchin.co.uk>
> Date: Sat, 05 Apr 2014 09:50:09 +0100
> 
> > If you try renaming a single file with "M-x rename-file RET", does it
> > also take 2 sec?
> >
> There's no apparent delay

On a restart or in a session where moving from Dired is already very
slow?  If the latter, then the actual file I/O is not the reason for
the slowdown, and I suggest to selectively disable portions of your
~/.emacs to find out what causes this.  It could be frequent GC, for
example (set garbage-collection-messages non-nil to see if it is), or
something else that causes excessive processing.  Or maybe you have a
lot of buffers visiting files in a directory being renamed, in which
case Dired will loop over all those files and rename their
default-directory, which could be expensive for a very large number of
such buffers.

Btw, I don't understand this part of your profile:

	- dired-in-this-tree                     10841  38%
	 + let                                      23   0%

This says that dired-in-this-tree takes 1/3rd of the run time.  But
dired-in-this-tree is just this:

  (defun dired-in-this-tree (file dir)
    ;;"Is FILE part of the directory tree starting at DIR?"
    (let (case-fold-search)
      (string-match-p (concat "^" (regexp-quote dir)) file)))

So I wonder how come it takes such a large proportion of the time.

As another possible hint, does the number of times dired-create-files
is called (26704) make sense?  Do you really have such a large number
of files in the tree you are moving?  Can you produce a profile for
moving a single file (which you say takes about 2 sec)?

Re: emacs slow on dired renaming

[Robert Marshall]

On Sat, Apr 05 2014, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Robert Marshall <spam@capuchin.co.uk>
>> Date: Sat, 05 Apr 2014 09:50:09 +0100
>> 
>> > If you try renaming a single file with "M-x rename-file RET", does it
>> > also take 2 sec?
>> >
>> There's no apparent delay
>
> On a restart or in a session where moving from Dired is already very
> slow?  If the latter, then the actual file I/O is not the reason for
> the slowdown, and I suggest to selectively disable portions of your
> ~/.emacs to find out what causes this.

In the session which is slow when using dired to rename

> It could be frequent GC, for
> example (set garbage-collection-messages non-nil to see if it is), or
> something else that causes excessive processing.  Or maybe you have a
> lot of buffers visiting files in a directory being renamed, in which
> case Dired will loop over all those files and rename their
> default-directory, which could be expensive for a very large number of
> such buffers.

The files which I'm renaming are all jpg's so there's no directories at
all in the file moving operation, could a soft link to that directory
have an effect (though in that case why doesn't it happen when emacs is
first started)?

>
> Btw, I don't understand this part of your profile:
>
> 	- dired-in-this-tree                     10841  38%
> 	 + let                                      23   0%
>
> This says that dired-in-this-tree takes 1/3rd of the run time.  But
> dired-in-this-tree is just this:
>
>   (defun dired-in-this-tree (file dir)
>     ;;"Is FILE part of the directory tree starting at DIR?"
>     (let (case-fold-search)
>       (string-match-p (concat "^" (regexp-quote dir)) file)))
>
> So I wonder how come it takes such a large proportion of the time.
>
> As another possible hint, does the number of times dired-create-files
> is called (26704) make sense?  Do you really have such a large number
> of files in the tree you are moving?  Can you produce a profile for
> moving a single file (which you say takes about 2 sec)?
>

This then (26704) - as you see from my comment about the jpg's which are
the only files being moved - doesn't appear to make any sense - I didn't
realise that number was the number of times the function was called. 10
files only are being moved none of which are directories, none of them
are links to other files

Here's moving a single file (using 'R' in a dired buffer) - the full trace rather than just a
selection from the report (again a jpg) from ~/Pictures to
~/Pictures/yy; 1083 calls to dired-do-rename(!). Could there be some
recursive calls somehow, why is command-execute being called 1227 times?!

- command-execute                                                1227  97%
 - call-interactively                                            1227  97%
  - dired-do-rename                                              1083  86%
   - dired-do-create-files                                       1083  86%
    - dired-create-files                                          962  76%
     - byte-code                                                  952  75%
      - dired-rename-file                                         766  61%
       - dired-rename-subdir                                      568  45%
        - dired-fun-in-all-buffers                                180  14%
           dired-buffers-for-dir                                  180  14%
       - dired-remove-file                                        194  15%
        - dired-fun-in-all-buffers                                194  15%
           dired-buffers-for-dir                                  186  14%
         - apply                                                    8   0%
          + dired-remove-entry                                      8   0%
      - dired-add-file                                            182  14%
       - dired-fun-in-all-buffers                                 182  14%
          dired-buffers-for-dir                                   178  14%
        - apply                                                     4   0%
         + dired-add-entry                                          4   0%
     + dired-file-marker                                           10   0%
    - dired-mark-read-file-name                                   107   8%
     - dired-mark-pop-up                                          107   8%
      - apply                                                     107   8%
       - read-file-name                                           107   8%
        - read-file-name-default                                  107   8%
         - completing-read                                         99   7%
          - completing-read-default                                99   7%
           - read-from-minibuffer                                  96   7%
            + timer-event-handler                                   1   0%
      dired-get-marked-files                                        4   0%
  - byte-code                                                     123   9%
   - read-extended-command                                        123   9%
    - completing-read                                             123   9%
     - completing-read-default                                    123   9%
      - read-from-minibuffer                                      108   8%
       - timer-event-handler                                        3   0%
        - byte-code                                                 3   0%
         - apply                                                    3   0%
            jit-lock-context-fontify                                2   0%
  + minibuffer-complete                                            15   1%
  + execute-extended-command                                        6   0%
- timer-event-handler                                              20   1%
 - byte-code                                                       13   1%
  - apply                                                          13   1%
   - which-func-update                                             12   0%
    - which-func-update-1                                          12   0%
     - byte-code                                                   12   0%
      - which-function                                             12   0%
       - add-log-current-defun                                     12   0%
          byte-code                                                 4   0%
   timer-activate                                                   3   0%
- redisplay_internal (C function)                                   7   0%
 - jit-lock-function                                                7   0%
  - jit-lock-fontify-now                                            7   0%
   - funcall                                                        7   0%
    - #<compiled 0xb436117>                                         7   0%
     - run-hook-with-args                                           7   0%
      - font-lock-fontify-region                                    7   0%
         font-lock-default-fontify-region                           7   0%
- ...                                                               0   0%
   Automatic GC                                                     0   0%

I've looked at pre-command-hook and post-command-hook and there doesn't
appear to be anything suspicious there (tooltip-hide and font-lock- fns)

In a session with no apparent slowdown, on a single dired-do-rename,
the function dired-do-rename gets called 42 times

Robert
-- 
La grenouille songe..dans son château d'eau
Links and things http://rmstar.blogspot.com/

Re: emacs slow on dired renaming

[Eli Zaretskii]

> From: Robert Marshall <spam@capuchin.co.uk>
> Date: Sat, 05 Apr 2014 12:09:42 +0100
> 
> > Btw, I don't understand this part of your profile:
> >
> > 	- dired-in-this-tree                     10841  38%
> > 	 + let                                      23   0%
> >
> > This says that dired-in-this-tree takes 1/3rd of the run time.  But
> > dired-in-this-tree is just this:
> >
> >   (defun dired-in-this-tree (file dir)
> >     ;;"Is FILE part of the directory tree starting at DIR?"
> >     (let (case-fold-search)
> >       (string-match-p (concat "^" (regexp-quote dir)) file)))
> >
> > So I wonder how come it takes such a large proportion of the time.
> >
> > As another possible hint, does the number of times dired-create-files
> > is called (26704) make sense?  Do you really have such a large number
> > of files in the tree you are moving?  Can you produce a profile for
> > moving a single file (which you say takes about 2 sec)?
> >
> 
> This then (26704) - as you see from my comment about the jpg's which are
> the only files being moved - doesn't appear to make any sense - I didn't
> realise that number was the number of times the function was called. 10
> files only are being moved none of which are directories, none of them
> are links to other files

No, wait, I think I was wrong -- this is the number of times the
profiler sampling found Emacs inside the function, not the number of
calls.  But that also sounds too large, as you say you were moving 10
files with 2 sec per file.  That should give 20000 samples (one sample
each millisecond), not 26700.

> Here's moving a single file (using 'R' in a dired buffer) - the full trace rather than just a
> selection from the report (again a jpg) from ~/Pictures to
> ~/Pictures/yy; 1083 calls to dired-do-rename(!). Could there be some
> recursive calls somehow, why is command-execute being called 1227 times?!
> 
> - command-execute                                                1227  97%

This is 1.2 sec, so it looks about OK.

>  - call-interactively                                            1227  97%
>   - dired-do-rename                                              1083  86%
>    - dired-do-create-files                                       1083  86%
>     - dired-create-files                                          962  76%
>      - byte-code                                                  952  75%
>       - dired-rename-file                                         766  61%
>        - dired-rename-subdir                                      568  45%
>         - dired-fun-in-all-buffers                                180  14%

According to this, dired-rename-subdir, dired-rename-file, and
dired-create-files took 70% of the time.  But I have no idea why.

Did you try turning on garbage-collection-messages?

If that doesn't give any clues, I guess the only way is to selectively
disable portions of your ~/.emacs, to find which part causes this.

Alternatively, try running "emacs -Q" for some time, and see if things
get slower there as well.

Re: emacs slow on dired renaming

[Robert Marshall]

On Sat, Apr 05 2014, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Robert Marshall <spam@capuchin.co.uk>
>> Date: Sat, 05 Apr 2014 12:09:42 +0100
>> 
>>  - call-interactively                                            1227  97%
>>   - dired-do-rename                                              1083  86%
>>    - dired-do-create-files                                       1083  86%
>>     - dired-create-files                                          962  76%
>>      - byte-code                                                  952  75%
>>       - dired-rename-file                                         766  61%
>>        - dired-rename-subdir                                      568  45%
>>         - dired-fun-in-all-buffers                                180  14%
>
> According to this, dired-rename-subdir, dired-rename-file, and
> dired-create-files took 70% of the time.  But I have no idea why.
>
> Did you try turning on garbage-collection-messages?
>

I didn't but have just tried it and there are no garbage collection messages

> If that doesn't give any clues, I guess the only way is to selectively
> disable portions of your ~/.emacs, to find which part causes this.
>
> Alternatively, try running "emacs -Q" for some time, and see if things
> get slower there as well.
>

I'm giving it a go with -Q and will report back - thanks!

Robert
-- 
La grenouille songe..dans son château d'eau
Links and things http://rmstar.blogspot.com/

Re: emacs slow on dired renaming

[Stefan Monnier]

>   (defun dired-in-this-tree (file dir)
>     ;;"Is FILE part of the directory tree starting at DIR?"
>     (let (case-fold-search)
>       (string-match-p (concat "^" (regexp-quote dir)) file)))

BTW, this is string-prefix-p, no?

> As another possible hint, does the number of times dired-create-files
> is called (26704) make sense?

This number is the number of samples, not the number of times the
function was called, right?


        Stefan

Re: emacs slow on dired renaming

[Robert Marshall]

On Sat, Apr 05 2014, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>>   (defun dired-in-this-tree (file dir)
>>     ;;"Is FILE part of the directory tree starting at DIR?"
>>     (let (case-fold-search)
>>       (string-match-p (concat "^" (regexp-quote dir)) file)))
>
> BTW, this is string-prefix-p, no?
>

Not in my copy of dired-in-this-tree where it is the same as that quoted
by Eli

Robert

Re: emacs slow on dired renaming

[Robert Marshall]

On Sat, Apr 05 2014, Robert Marshall <spam@capuchin.co.uk> wrote:

> On Sat, Apr 05 2014, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> If that doesn't give any clues, I guess the only way is to selectively
>> disable portions of your ~/.emacs, to find which part causes this.
>>
>> Alternatively, try running "emacs -Q" for some time, and see if things
>> get slower there as well.
>>
>
> I'm giving it a go with -Q and will report back - thanks!
>
I've had 2 sessions of emacs running for the past 24 hours, one with -Q
and I still see no slowdown there, the other run with --no-desktop in
which I ran VM (which I use for email and where I originally saw a
slowness problem - which *may* be related to the problem I then found in
dired) there also there's no slowness. In the -Q session I did very
little other than an occasional test of dired speed. In the other
session I only ran VM and dired.

So it would appear to be either what I'm loading in .emacs.desktop or
what I'm doing after that in that session which causes that symptom to
gradually appear. I've never seen the problem with an emacs session
which lasts less than c 2 hours.

Robert
-- 
La grenouille songe..dans son château d'eau

Re: emacs slow on dired renaming

[Eli Zaretskii]

> From: Robert Marshall <spam@capuchin.co.uk>
> Date: Sun, 06 Apr 2014 13:39:11 +0100
> 
> I've had 2 sessions of emacs running for the past 24 hours, one with -Q
> and I still see no slowdown there, the other run with --no-desktop in
> which I ran VM (which I use for email and where I originally saw a
> slowness problem - which *may* be related to the problem I then found in
> dired) there also there's no slowness. In the -Q session I did very
> little other than an occasional test of dired speed. In the other
> session I only ran VM and dired.
> 
> So it would appear to be either what I'm loading in .emacs.desktop or
> what I'm doing after that in that session which causes that symptom to
> gradually appear. I've never seen the problem with an emacs session
> which lasts less than c 2 hours.

I guess the next step is to look in your desktop file.

Re: emacs slow on dired renaming

[Robert Marshall]

On Sun, Apr 06 2014, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Robert Marshall <spam@capuchin.co.uk>
>> Date: Sun, 06 Apr 2014 13:39:11 +0100
>> So it would appear to be either what I'm loading in .emacs.desktop or
>> what I'm doing after that in that session which causes that symptom to
>> gradually appear. I've never seen the problem with an emacs session
>> which lasts less than c 2 hours.
>
> I guess the next step is to look in your desktop file.
>

First I'm going to try starting an emacs session with the .emacs.desktop
file loaded and largely leave that session alone (apart from dired
renames) give it 12 hours and see if the problem appears. If so I'll
start reducing the desktop, otherwise I'm going to have to look at what I
do in emacs in the normal session -  I'll compare featurep between the
desktop left alone session and now. 

Robert
-- 
La grenouille songe..dans son château d'eau

Re: emacs slow on dired renaming

[Robert Marshall]

On Sun, Apr 06 2014, Robert Marshall <spam@capuchin.co.uk> wrote:

> On Sun, Apr 06 2014, Eli Zaretskii <eliz@gnu.org> wrote:
>
>>> From: Robert Marshall <spam@capuchin.co.uk>
>>> Date: Sun, 06 Apr 2014 13:39:11 +0100
>>> So it would appear to be either what I'm loading in .emacs.desktop or
>>> what I'm doing after that in that session which causes that symptom to
>>> gradually appear. I've never seen the problem with an emacs session
>>> which lasts less than c 2 hours.
>>
>> I guess the next step is to look in your desktop file.
>>
>
> First I'm going to try starting an emacs session with the .emacs.desktop
> file loaded and largely leave that session alone (apart from dired
> renames) give it 12 hours and see if the problem appears. If so I'll
> start reducing the desktop, otherwise I'm going to have to look at what I
> do in emacs in the normal session -  I'll compare featurep between the
> desktop left alone session and now. 
>

That session - which I left sitting there didn't produce the problem.
AFAICT, there appears to be a point where the memory footprint of the bad
emacs session greatly increases (but spotting when it does is still
eluding me)

/proc/$emacspid/status is currently giving me

VmPeak:	 2123196 kB
VmSize:	 2120600 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	 1274504 kB
VmRSS:	  983744 kB
VmData:	 1373744 kB
VmStk:	     156 kB
VmExe:	    2184 kB
VmLib:	   36492 kB
VmPTE:	    2816 kB
VmSwap:	  370112 kB

the process is using around 25% of the available memory (4GB) but maybe
the problem is that there isn't more and it's spending its time getting
out of swap?

I've not opened any particularly large buffers since I started this session
(some were opened during startup by the desktop but the largest one
is around 1meg)
Looking at the variables suggested by
https://www.gnu.org/software/emacs/manual/html_node/elisp/Memory-Usage.html

cons-cells-consed
59741973
floats-consed
196763
vector-cells-consed
71162058
symbols-consed
252132
string-chars-consed
307585551
misc-objects-consed
1241806
intervals-consed
400985
strings-consed
7853976

do any of these look out of the ordinary for an emacs session which has
been running around 24 hours (having suspended the machine overnight)?

A manual call to garbage-collect returns nil and pure-bytes-used is 88 -
I don't know what this should be but it doesn't look sensible? Maybe a
refresh from bzr and rebuild?

Robert
-- 
La grenouille songe..dans son château d'eau

Re: emacs slow on dired renaming

[Robert Marshall]

On Wed, Apr 09 2014, Robert Marshall <spam@capuchin.co.uk> wrote:

>
> A manual call to garbage-collect returns nil and pure-bytes-used is 88 -
> I don't know what this should be but it doesn't look sensible? Maybe a
> refresh from bzr and rebuild?
>

Is this another instance of bug 17168 - and as I'm on bzr trunk do I
need to try applying the patch in that thread?

Robert
-- 
La grenouille songe..dans son château d'eau

Re: emacs slow on dired renaming

[Stefan Monnier]

> A manual call to garbage-collect returns nil

Quoting C-h garbage-collect:

   However, if there was overflow in pure space, `garbage-collect'
   returns nil, because real GC can't be done.

> and pure-bytes-used is 88 -

That indicates that pure space was overflowed.  IOW your Emacs build
is broken.  You need to rebuild with a larger pure space.


        Stefan

Re: emacs slow on dired renaming

[Robert Marshall]

On Wed, Apr 09 2014, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> A manual call to garbage-collect returns nil
>
> Quoting C-h garbage-collect:
>
>    However, if there was overflow in pure space, `garbage-collect'
>    returns nil, because real GC can't be done.
>
>> and pure-bytes-used is 88 -
>
> That indicates that pure space was overflowed.  IOW your Emacs build
> is broken.  You need to rebuild with a larger pure space.
>

Yes, I updated from bzr and that build gave me the
pure-space-overflow-message on startup (which the previous build didn't)
so I altered SYSTEM_PURESIZE_EXTRA and things are - so far - looking
good!

Robert
-- 
La grenouille songe..dans son château d'eau