[O] Help speeding up Org iCal export

[O] Help speeding up Org iCal export

[Ken Mankoff]

Hello,

When I eval `(org-icalendar-combine-agenda-files nil)`, it takes about 15 seconds.

Some strange messages in the *Messages* buffer include:

22 files scanned, 7 files contains IDs and in total 23 IDs found.

But my `org-agenda-files` variable only contains 12 files and no #+INCLUDE statements.

There are also several dozen lines of

org-babel-exp process sh at position 74770...
org-babel-exp process sh at position 74830...
org-babel-exp process sh at position 75147...
...
org-babel-exp process sqlite at position 27700...
org-babel-exp process sqlite at position 27704...
org-babel-exp process sh at position 27604...
org-babel-exp process sh at position 27617...
org-babel-exp process sh at position 27691...
org-babel-exp process translate at position 27970...

(and with bash, python, emacs-lisp, and other languages). When I view for example the only #BEGIN_SRC translate block in any of my files it is just a test in my notes file:

#+BEGIN_SRC translate :src en :dest da
This is a test
#+END_SRC

I cannot eval it with C-c C-c because I have set ":eval no" system-wide with:

  (setq org-babel-default-header-args
	(cons '(:eval . "no")
	      (assq-delete-all :eval org-babel-default-header-args)))


Sometimes the Messages contain confirmation that a code block is disabled:

org-babel-exp process plantuml at position 27719...
Evaluation of this plantuml code block is disabled.
org-babel-exp process plantuml at position 27762...
Evaluation of this plantuml code block is disabled.


But those blocks don't have any special header:

#+begin_src plantuml :file tryout.png
...misc plantuml code here.
#+end_src


Can anyone offer suggestions why it reports scanning 22 files, and how to speed it up since no code should be evaluated?

Thank you,

  -k.

Re: [O] Help speeding up Org iCal export

[Eric S Fraga]

On Wednesday, 27 May 2020 at 17:24, Ken Mankoff wrote:
> Can anyone offer suggestions why it reports scanning 22 files, and how
> to speed it up since no code should be evaluated?

Anecdotally, I have the impression that all org buffers in the current
Emacs instance are scanned as well as the files noted in the agenda
list.  But I have no proof of this nor do I know why it would do
this.  I have the same issue with org-caldav-sync.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.3.6-640-g9bc0cc

Re: Help speeding up Org iCal export

[Nicolas Goaziou]

Hello,

I see a few misconceptions here.

Ken Mankoff <mankoff@gmail.com> writes:

> When I eval `(org-icalendar-combine-agenda-files nil)`, it takes about
> 15 seconds.

OK. You may want to profile this, or use ELP.

> Some strange messages in the *Messages* buffer include:
>
> 22 files scanned, 7 files contains IDs and in total 23 IDs found.

You may have set `org-icalendar-store-UID' somehow.

> But my `org-agenda-files` variable only contains 12 files and no
> #+INCLUDE statements.

ID scanning is not limited to agenda files. Org also scans files known
to contain ID. E.g., if, during the session, you insert an ID in
"foo.org", later scans (in the same session) will include "foo.org".

See `org-id-files' (internal variable), and `org-id-extra-files', which
may contain even more files.

> There are also several dozen lines of
>
> org-babel-exp process sh at position 74770...
> org-babel-exp process sh at position 74830...
> org-babel-exp process sh at position 75147...
> ...
> org-babel-exp process sqlite at position 27700...
> org-babel-exp process sqlite at position 27704...
> org-babel-exp process sh at position 27604...
> org-babel-exp process sh at position 27617...
> org-babel-exp process sh at position 27691...
> org-babel-exp process translate at position 27970...
>
> (and with bash, python, emacs-lisp, and other languages). When I view for example the only #BEGIN_SRC translate block in any of my files it is just a test in my notes file:
>
> #+BEGIN_SRC translate :src en :dest da
> This is a test
> #+END_SRC
>
>
> I cannot eval it with C-c C-c because I have set ":eval no"
> system-wide with:

This is a red herring, IMO. The message above means the block is being
processed for export, not necessarily evaluated. In order to know how to
handle the block, Org needs to process it.

>   (setq org-babel-default-header-args
> 	(cons '(:eval . "no")
> 	      (assq-delete-all :eval org-babel-default-header-args)))
>
>
> Sometimes the Messages contain confirmation that a code block is disabled:
>
> org-babel-exp process plantuml at position 27719...
> Evaluation of this plantuml code block is disabled.
> org-babel-exp process plantuml at position 27762...
> Evaluation of this plantuml code block is disabled.

Here, it tries to evaluate the blocks, obviously, but eventually
doesn't, per your settings.

> But those blocks don't have any special header:
>
> #+begin_src plantuml :file tryout.png
> ...misc plantuml code here.
> #+end_src

I see no issue here. Org checks if it should evaluate it, and cannot.

> Can anyone offer suggestions why it reports scanning 22 files, and how
> to speed it up since no code should be evaluated?

See above. Instead of wild guesses, you could use an appropriate tool.

Regards,

-- 
Nicolas Goaziou

Re: Help speeding up Org iCal export

[Ken Mankoff]

[-- Attachment #1: Type: text/plain, Size: 3835 bytes --]

Hi Nicolas,

Thank you for providing suggestions about the appropriate tool. I've
profiled it, examined the variables you suggested, but am not sure what
next steps to take. I hope it is OK that I post questions about this on
this list...

On Thu, May 28, 2020 at 5:17 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Ken Mankoff <mankoff@gmail.com> writes:
>
> > When I eval `(org-icalendar-combine-agenda-files nil)`, it takes about
> > 15 seconds.
>
> OK. You may want to profile this, or use ELP.
>

Output from the profiler is here: https://pastebin.com/QvL9w69R but the key
lines seems to be:

      - org-icalendar-combine-agenda-files                      15540  83%
       - apply                                                  15540  83%
        - org-icalendar--combine-files                          15540  83%
         - mapconcat                                            15536  83%
          - #<compiled 0x15879b1>                               15536  83%
           - org-export-as                                      15162  81%
            - org-export-data                                    6493  34%
             - mapconcat                                         6493  34%
              - #<compiled 0x25baf9d>                            6493  34%
               - org-export-data                                 6493  34%
                - mapconcat                                      6164  33%
                 - #<compiled 0x22864b5>                         6160  33%
                  - org-export-data                              6156  33%
                   - mapconcat                                   5674  30%

with org-export-data and mapconcat repeating, AND

                        - mapc                                   5672  30%
                         - #<compiled 0x3129c91>                 5672  30%
                          - mapc                                 5672  30%
                           - #<compiled 0x3129c91>               5668  30%
                            - #<compiled 0x15b2141>               4392  23%
                             - org-id-find                       4392  23%
                              - org-id-update-id-locations
4376  23%
                               - org-map-entries                 3696  19%
                                - org-scan-tags                  3668  19%
                                 + #<lambda 0x22c5233ea>                990
  5%
                                 + #<lambda 0x22c5233ea>                879
  4%
                                 + #<lambda 0x22c5233ea>                830
  4%



> Some strange messages in the *Messages* buffer include:

> >
> > 22 files scanned, 7 files contains IDs and in total 23 IDs found.
>
> You may have set `org-icalendar-store-UID' somehow.
>

It is set to nil. I have only the *scratch* buffer open, and no other
buffers (although some where open earlier in this emacs session).


> > But my `org-agenda-files` variable only contains 12 files and no
> > #+INCLUDE statements.
>
> ID scanning is not limited to agenda files. Org also scans files known
> to contain ID. E.g., if, during the session, you insert an ID in
> "foo.org", later scans (in the same session) will include "foo.org".
>
> See `org-id-files' (internal variable), and `org-id-extra-files', which
> may contain even more files.
>

`org-id-files` contained a subset of my agenda files plus a few extra
(total: 7 files, and these 7 plus my agenda files do not sum to 22). When I
set it to nil, nothing significant changed, but it was reset after running
`org-icalendar-combine-agenda-files` again.

`org-id-extra-files` was set to org-agenda-text-search-extra-files. When I
set it to nil, nothing significant changed. It does not get reset when
`org-icalendar-combine-agenda-files` is run.

  -k.

[-- Attachment #2: Type: text/html, Size: 6256 bytes --]

Re: Help speeding up Org iCal export

[Josh Moller-Mara]

I'm running into this issue now.

One thing I noticed is that if you have broken links (and
org-export-with-broken-links is t), org-id-update-id-locations will be
run for every broken link it finds.  This turns out to be quite slow and
unnecessary since no files will have changed in between
org-id-update-id-locations runs during export.

One hacky workaround is to temporarily disable
org-id-update-id-locations, like so:

(cl-letf (((symbol-function 'org-id-update-id-locations)
	     (lambda (&optional files silent) (message "Not updating locations"))))
    (let ((org-export-with-broken-links t)
	  (gc-cons-threshold 80000000))
      (org-icalendar-combine-agenda-files)))

The right solution would probably be to not have broken links in the
first place. :P
Or maybe org-id-update-id-locations could be debounced or cached in some way.

Best,
Josh