Re: Exporting large documents

[Lawrence Mitchell <lawrence.mitchell@ed.ac.uk>]

[Reintroducing org mailing list CC]

On 05/05/2013 20:21, Nicolas Goaziou wrote:
> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
>>> I don't think there's much to do about that. Though, some tools could
>>> benefit from caching, like Lawrence did for
>>> `org-export-resolve-fuzzy-link'.
>>
>> Could you point out specific ones where it would make sense?  Maybe
>> someone would like to take this up as a task.
>
> It requires some careful benchmarking. Though, good candidates are tools
> searching for an object or element within the full parse tree. This
> includes:
>
>    - org-export-footnote-first-reference-p
>    - org-export-get-footnote-number
>    - org-export-get-category
>    - org-export-resolve-coderef
>    - org-export-resolve-radio-link
>    - org-export-get-loc
>    - org-export-get-ordinal

I wonder if it would be possible to store a copy of the parse tree in a 
form that is more amenable to log or constant time searches.

However, I note that my caching of the fuzzy link stuff brought the 
quadratic time export of copies of the org manual introduction down to 
linear (or close to) time.

Most of the problem now seems to be that for big documents, many 
functions are called a /lot/.  For example:

org-element--current-element takes (on my machine) 0.0003 seconds per 
call.  However, when exporting 128x the orgmanual introduction, it's 
called around 250000 times giving ~ 80 seconds total time (out of ~200 
total).

So it sort of feels like actually what is needed is microoptimisations 
of the bits of the export engine that are called the most.

Lawrence


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.