Bug?/performance issue in org-icalendar-export-current-agenda

Bug?/performance issue in org-icalendar-export-current-agenda

[Richard Lawrence]

Hi all,

I pulled from master recently (for the first time since the spring), and
since the update, I have noticed a performance/lockup issue.  The
problem occurs when exporting an agenda view to iCalendar format.

My Emacs version is 23.4.1 and my Org version is "8.3beta" (I am
currently on commit 767895...)
 
I have traced the problem down to a call to
`org-icalendar-export-current-agenda', though I don't know if the
problem is in this function or in another function called by it.
Somewhere during the execution of this function, Emacs goes to 100% CPU
usage (on one core) and memory usage starts growing until I kill the
process.
 
Here's what my setup looks like.  I have just one entry in
`org-agenda-custom-commands' that defines the files field.  It looks
like:

 ("Z" "Export to iCalendar file" tags "+event|+appointment" nil
  ("~/Documents/website/public_html/lib/attachments/calendar.ics")

I normally only call this command via a cron job that exports my agenda
files to .ics, which until recently was working fine.  But I have
noticed I can reproduce the lockup interactively by building an agenda
view with this command, and then calling
`org-icalendar-export-current-agenda' on it (which is also what happens,
a few steps down the call chain, during the cron job).

I have run some tests by manually setting org-agenda-files, then
building this agenda and eval'ing

(org-icalendar-export-current-agenda "/tmp/agenda.ics") 

in the agenda buffer.  The results:

1) Using just a simple three-entry test file, the export completes quickly
   and I don't see the lockup.  
2) Using an agenda file which contains plenty of real data
   (about 1000 lines) but no headlines tagged "event" or "appointment"
   (i.e., a file for which this agenda view is empty), the call to
   `org-icalendar-export-current-agenda' takes about 5 seconds to
   complete, even though there are no entries to export.  During those
   5 seconds, the Emacs process goes to 100% CPU usage.
3) Using a different agenda file (of about 4000 lines) which contains
   about 75 matching headlines, the call to
   `org-icalendar-export-current-agenda' does not complete within
   several minutes, during which time I see 100% CPU usage and rapid
   memory usage growth.  

So it looks to me like maybe some part of the export process uses
exponentially more time and memory, depending on the input, since a
minimal test case does not produce the problem.

What's the next step for debugging this?  Please let me know if I can
provide more information.  

Thanks! 

-- 
Best,
Richard

Re: Bug?/performance issue in org-icalendar-export-current-agenda

[OSiUX]

[-- Attachment #1: Type: text/plain, Size: 3856 bytes --]

El mar, 26 ago 2014, Richard Lawrence decía:

> Hi all,
> 
> I pulled from master recently (for the first time since the spring), and
> since the update, I have noticed a performance/lockup issue.  The
> problem occurs when exporting an agenda view to iCalendar format.
> 
> My Emacs version is 23.4.1 and my Org version is "8.3beta" (I am
> currently on commit 767895...)
>  
> I have traced the problem down to a call to
> `org-icalendar-export-current-agenda', though I don't know if the
> problem is in this function or in another function called by it.
> Somewhere during the execution of this function, Emacs goes to 100% CPU
> usage (on one core) and memory usage starts growing until I kill the
> process.
>  
> Here's what my setup looks like.  I have just one entry in
> `org-agenda-custom-commands' that defines the files field.  It looks
> like:
> 
>  ("Z" "Export to iCalendar file" tags "+event|+appointment" nil
>   ("~/Documents/website/public_html/lib/attachments/calendar.ics")
> 
> I normally only call this command via a cron job that exports my agenda
> files to .ics, which until recently was working fine.  But I have
> noticed I can reproduce the lockup interactively by building an agenda
> view with this command, and then calling
> `org-icalendar-export-current-agenda' on it (which is also what happens,
> a few steps down the call chain, during the cron job).
> 
> I have run some tests by manually setting org-agenda-files, then
> building this agenda and eval'ing
> 
> (org-icalendar-export-current-agenda "/tmp/agenda.ics") 
> 
> in the agenda buffer.  The results:
> 
> 1) Using just a simple three-entry test file, the export completes quickly
>    and I don't see the lockup.  
> 2) Using an agenda file which contains plenty of real data
>    (about 1000 lines) but no headlines tagged "event" or "appointment"
>    (i.e., a file for which this agenda view is empty), the call to
>    `org-icalendar-export-current-agenda' takes about 5 seconds to
>    complete, even though there are no entries to export.  During those
>    5 seconds, the Emacs process goes to 100% CPU usage.
> 3) Using a different agenda file (of about 4000 lines) which contains
>    about 75 matching headlines, the call to
>    `org-icalendar-export-current-agenda' does not complete within
>    several minutes, during which time I see 100% CPU usage and rapid
>    memory usage growth.  
> 
> So it looks to me like maybe some part of the export process uses
> exponentially more time and memory, depending on the input, since a
> minimal test case does not produce the problem.
>
> What's the next step for debugging this?  Please let me know if I can
> provide more information.  
> 
> Thanks! 
> 
> -- 
> Best,
> Richard

I use a bash script: =agenda-ascii= [0] to
reduce the list of file to export, first =grep=
current =<year-month= pattern in the *.org
files:

  #+BEGIN_EXAMPLE
    ORG_FILES=$($GREP -l "<$YEAR-$MONTH" *.org)
    echo $ORG_FILES

    if [ "$EXPORT_ICS" = "TRUE" ]
    then
        for AGENDA in $(echo $ORG_FILES)
        do
            echo EXPORT $AGENDA ...
            ARCHIVO=$DIR/$AGENDA
            $EMACS --batch -q --directory=${DIR} --visit=${ARCHIVO} --eval '(org-export-icalendar-this-file)' >/dev/null 2&>1
        done
    fi
  #+END_EXAMPLE

Actually the process, cosume 42%-97% CPU, for
526 appoinments, takes 0m 30.396s

  Org-mode version 8.3beta (release_8.3beta-264-g7a5f3f
  GNU Emacs 24.3.1 (x86_64-pc-linux-gnu) of 2014-06-06 on barber, modified by Debian


[0] https://github.com/osiris/org-bash-utils/blob/master/agenda-ascii

-- 

::

  Osiris Alejandro Gomez (OSiUX) osiux@osiux.com.ar
  DC44 95D2 0D5D D544 FC1A F00F B308 A671 9237 D36C
  http://www.osiux.com.ar http://www.altermundi.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Bug?/performance issue in org-icalendar-export-current-agenda

[Nicolas Goaziou]

Hello,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> I pulled from master recently (for the first time since the spring), and
> since the update, I have noticed a performance/lockup issue.  The
> problem occurs when exporting an agenda view to iCalendar format.
>
> My Emacs version is 23.4.1 and my Org version is "8.3beta" (I am
> currently on commit 767895...)
>  
> I have traced the problem down to a call to
> `org-icalendar-export-current-agenda', though I don't know if the
> problem is in this function or in another function called by it.
> Somewhere during the execution of this function, Emacs goes to 100% CPU
> usage (on one core) and memory usage starts growing until I kill the
> process.

Interestingly, I introduced a patch that should speed up this function
yesterday. You may want to update Org (the patch landed after 767895)
and try again.

If it is still suspiciously slow, please use profiler (`profiler-start')
and report the results here.


Regards,

-- 
Nicolas Goaziou

Re: Bug?/performance issue in org-icalendar-export-current-agenda

[Richard Lawrence]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Interestingly, I introduced a patch that should speed up this function
> yesterday. You may want to update Org (the patch landed after 767895)
> and try again.

Indeed!  I just pulled again and the problem is gone.  The export runs
much more reasonably now, completing in a few seconds even with all my
agenda files in the mix.  Thanks!

Best,
Richard