Performance problem

Performance problem

[Michael Welle]

Hallo,

I use Emacs 23.2.1 and org-mode 7.5. After some time, usually after a
few days of 'uptime', building an agenda becomes very sloooow. I expect
the agenda built from 120 org file to show up within five seconds. If
Emacs is in the bad state it takes three minutes. Restarting Emacs
solves the problem. 

I don't blame org-mode for that behaviour, but maybe someone of you has
observed such behaviour, too? Any hints on debugging? Building the
agenda results in more than 60k stat syscalls. Mostly /etc/localtime is
stat'ted and the org files. That is strange, but not different if all
works as expected.

Regards
hmw


-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

Re: Performance problem

[Matt Lundin]

Michael Welle <mwe012008@gmx.net> writes:

> I use Emacs 23.2.1 and org-mode 7.5. After some time, usually after a
> few days of 'uptime', building an agenda becomes very sloooow. 

Is this the standard agenda or a custom agenda?

> I expect the agenda built from 120 org file to show up within five
> seconds. 

Five seconds still seems like a long time. How many lines are in your
agenda? Are your org buffers already open when you call it? Or have you
closed your org buffers by typing "x" in the agenda?

> If Emacs is in the bad state it takes three minutes. Restarting Emacs
> solves the problem.

> I don't blame org-mode for that behaviour, but maybe someone of you has
> observed such behaviour, too? Any hints on debugging? 

M-x elp-instrument-package org [RET]
M-x org-agenda-list
M-x elp-results

Best,
Matt

> Building the agenda results in more than 60k stat syscalls. Mostly
> /etc/localtime is stat'ted and the org files. That is strange, but not
> different if all works as expected.

Re: Performance problem

[Michael Welle]

Hi,

Matt Lundin <mdl@imapmail.org> writes:

> Michael Welle <mwe012008@gmx.net> writes:
>
>> I use Emacs 23.2.1 and org-mode 7.5. After some time, usually after a
>> few days of 'uptime', building an agenda becomes very sloooow. 
>
> Is this the standard agenda or a custom agenda?
it's a custom agenda that collects all appointments for the next four
weeks:

       ( "z" "Appointments for the next four weeks" agenda ""
        ((org-agenda-skip-function
          (lambda nil
        (org-agenda-skip-entry-if 'nottodo '("APPT"))))
         (org-agenda-ndays 28)
         (org-agenda-include-diary nil )))


>> I expect the agenda built from 120 org file to show up within five
>> seconds. 
>
> Five seconds still seems like a long time. How many lines are in your
> agenda? Are your org buffers already open when you call it? Or have you
> closed your org buffers by typing "x" in the agenda?
OK, five seconds is an upper boundary, it might take more like three
seconds or so. I count twentyone, twentytwo, ... to 'measure' the
duration of the operation. With this higly accurate method I can't
observe much difference if I started with loaded org files or not.


>> If Emacs is in the bad state it takes three minutes. Restarting Emacs
>> solves the problem.
>
>> I don't blame org-mode for that behaviour, but maybe someone of you has
>> observed such behaviour, too? Any hints on debugging? 
>
> M-x elp-instrument-package org [RET]
> M-x org-agenda-list
> M-x elp-results
Ah, nice. Now I have to wait til the misbehaviour occurs again.

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

Re: Performance problem

[Michael Welle]

Hi,

Michael Welle <mwe012008@gmx.net> writes:
[...]
>>> I don't blame org-mode for that behaviour, but maybe someone of you has
>>> observed such behaviour, too? Any hints on debugging? 
>>
>> M-x elp-instrument-package org [RET]
>> M-x org-agenda-list
>> M-x elp-results
> Ah, nice. Now I have to wait til the misbehaviour occurs again.
a different machine, but the same problem ;).  This is a 'bad' run 


org-agenda-list                   1           17.646198     17.646198
org-agenda-get-day-entries        868         12.302421999  0.0141732972
org-agenda-get-sexps              868         11.237522999  0.0129464550
org-prepare-agenda-buffers        2           4.5861040000  2.2930520000
org-prepare-agenda                1           4.524422      4.524422
org-get-agenda-file-buffer        1116        4.2117990000  0.0037740134
org-mode                          124         1.5946210000  0.0128598467
org-install-agenda-files-menu     124         1.0290419999  0.0082987258
org-agenda-files                  134         0.84026       0.0062705970
org-find-base-buffer-visiting     1116        0.6872910000  0.0006158521
org-get-entries-from-diary        7           0.4210619999  0.0601517142
org-agenda-get-blocks             868         0.3384840000  0.0003899585


And this is a 'good' run

org-agenda-list                   1           3.935774      3.935774
org-prepare-agenda-buffers        2           2.894058      1.447029
org-prepare-agenda                1           2.833592      2.833592
org-get-agenda-file-buffer        1116        2.5790829999  0.0023110062
org-mode                          124         1.1269950000  0.0090886693
org-install-agenda-files-menu     124         0.8238960000  0.0066443225
org-agenda-files                  134         0.6465149999  0.0048247388
org-agenda-get-day-entries        868         0.5411369999  0.0006234297
org-get-entries-from-diary        7           0.49768       0.0710971428
org-find-base-buffer-visiting     1116        0.3149529999  0.0002822159
org-set-startup-visibility        124         0.1660580000  0.0013391774
org-refresh-category-properties   248         0.1442590000  0.0005816895

Looks like it spent most of the extra time in org-agenda-get-day-entries
and org-agenda-get-sexps. Time to have a look at the source code ;).

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

Re: Performance problem

[Eric S Fraga]

Michael Welle <mwe012008@gmx.net> writes:

> Hi,
>
> Matt Lundin <mdl@imapmail.org> writes:
>
>> Michael Welle <mwe012008@gmx.net> writes:
>>
>>> I use Emacs 23.2.1 and org-mode 7.5. After some time, usually after a
>>> few days of 'uptime', building an agenda becomes very sloooow. 
>>
>> Is this the standard agenda or a custom agenda?
> it's a custom agenda that collects all appointments for the next four
> weeks:
>
>        ( "z" "Appointments for the next four weeks" agenda ""
>         ((org-agenda-skip-function
>           (lambda nil
>         (org-agenda-skip-entry-if 'nottodo '("APPT"))))
>          (org-agenda-ndays 28)
>          (org-agenda-include-diary nil )))
>
>
>>> I expect the agenda built from 120 org file to show up within five
>>> seconds. 
>>
>> Five seconds still seems like a long time. How many lines are in your
>> agenda? Are your org buffers already open when you call it? Or have you
>> closed your org buffers by typing "x" in the agenda?
> OK, five seconds is an upper boundary, it might take more like three
> seconds or so. I count twentyone, twentytwo, ... to 'measure' the
> duration of the operation. With this higly accurate method I can't
> observe much difference if I started with loaded org files or not.

Well, for me, a monthly view (equivalent to your 4 week view?)
takes well over 5 seconds at best of times:

--8<---------------cut here---------------start------------->8---
org-agenda-list                                               2           19.049079     9.5245395
org-agenda-view-mode-dispatch                                 1           18.791959     18.791959
org-agenda-month-view                                         1           18.529849     18.529849
org-agenda-change-time-span                                   1           18.529803     18.529803
org-agenda-redo                                               1           18.528674     18.528674
org-agenda-get-day-entries                                    462         18.115907999  0.0392119220
org-let                                                       1           17.600798     17.600798
org-agenda-get-scheduled                                      462         9.5824770000  0.0207412922
org-get-todo-state                                            12382       6.0307460000  0.0004870575
org-back-to-heading                                           13407       4.3458889999  0.0003241507
org-agenda-get-timestamps                                     462         3.3732049999  0.0073013095
org-agenda-get-deadlines                                      462         2.6114349999  0.0056524567
org-agenda-skip                                               13809       1.8835389999  0.0001363993
org-time-string-to-absolute                                   11880       1.5394879999  0.0001295865
org-agenda-get-blocks                                         462         1.4670860000  0.0031755108
org-prepare-agenda                                            2           1.2731379999  0.6365689999
--8<---------------cut here---------------end--------------->8---

but this is on a netbook...:

,----
| processor	: 0
| vendor_id	: GenuineIntel
| cpu family	: 6
| model		: 28
| model name	: Intel(R) Atom(TM) CPU N450   @ 1.66GHz
| stepping	: 10
| cpu MHz		: 1667.000
| cache size	: 512 KB
| physical id	: 0
| siblings	: 2
`----

this explains why I don't use a monthly view very often... ;-)

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.288.gcec8)

Re: Performance problem

[Michael Welle]

Hi,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Michael Welle <mwe012008@gmx.net> writes:
[...]
>>>> I expect the agenda built from 120 org file to show up within five
>>>> seconds. 
>>>
>>> Five seconds still seems like a long time. How many lines are in your
>>> agenda? Are your org buffers already open when you call it? Or have you
>>> closed your org buffers by typing "x" in the agenda?
>> OK, five seconds is an upper boundary, it might take more like three
>> seconds or so. I count twentyone, twentytwo, ... to 'measure' the
>> duration of the operation. With this higly accurate method I can't
>> observe much difference if I started with loaded org files or not.
>
> Well, for me, a monthly view (equivalent to your 4 week view?)
> takes well over 5 seconds at best of times:
[...]
> but this is on a netbook...:
>
> ,----
> | processor	: 0
> | vendor_id	: GenuineIntel
> | cpu family	: 6
> | model		: 28
> | model name	: Intel(R) Atom(TM) CPU N450   @ 1.66GHz
> | stepping	: 10
> | cpu MHz		: 1667.000
> | cache size	: 512 KB
> | physical id	: 0
> | siblings	: 2
> `----
>
> this explains why I don't use a monthly view very often... ;-)
;) 

That is my most used agenda view. It shows my appointments and I use it
every time I make an appointment. My machine is a little bit faster than
yours:

model name      : Intel(R) Pentium(R) Dual  CPU  T2390  @ 1.86GHz

The agenda view could be faster, of course, especially if a customer is
at the phone for an appointment and you have to scroll back and forth a
few times. But it works.

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

Re: Performance problem

[Matt Lundin]

Michael Welle <mwe012008@gmx.net> writes:

> Matt Lundin <mdl@imapmail.org> writes:
>
>> Michael Welle <mwe012008@gmx.net> writes:
>>
>>> I use Emacs 23.2.1 and org-mode 7.5. After some time, usually after a
>>> few days of 'uptime', building an agenda becomes very sloooow.
>>
>> Is this the standard agenda or a custom agenda?
> it's a custom agenda that collects all appointments for the next four
> weeks:
>
>        ( "z" "Appointments for the next four weeks" agenda ""
>         ((org-agenda-skip-function
>           (lambda nil
>         (org-agenda-skip-entry-if 'nottodo '("APPT"))))
>          (org-agenda-ndays 28)
>          (org-agenda-include-diary nil )))

A monthly view does indeed take a long time to appear. Skip functions
are very useful but, alas, not particularly efficient. This command
first considers all entries (SCHEDULED, DEADLINE, timestamp) that fall
within the next 28 days and then selects only those entries with the
keyword APPT.

Might I ask whether your appointments are all timestamps or diary sexps
(as opposed to SCHEDULED or DEADLINE entries)? If so, adding the
following setting to the command may speed things up a bit:

(org-agenda-entry-types '(:timestamp :sexp))

Best,
Matt