From mboxrd@z Thu Jan  1 00:00:00 1970
From: Lawrence Mitchell <lawrence.mitchell@ed.ac.uk>
Subject: Re: Exporting large documents
Date: Mon, 06 May 2013 12:07:53 +0100
Message-ID: <51878F09.1050904@ed.ac.uk>
References: <877gjnojsq.fsf@Rainer.invalid>
	<5654CA29-5F6D-4E8B-8B8B-C3609D76D189@gmail.com>
	<8761z5gw6w.fsf@gmx.li>
	<707EAAA5-D27C-47B7-9A1E-874C3A375BD9@gmail.com>
	<87zjwcwc4b.fsf@gmx.li> <877gjfgnl9.fsf@gmail.com>
	<0F877AB5-D488-4223-B0E7-F11B4B973614@gmail.com>
	<87ip2xfd0x.fsf@gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Return-path: <emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org>
Received: from eggs.gnu.org ([208.118.235.92]:41468)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <lawrence.mitchell@ed.ac.uk>) id 1UZJHL-0004R7-Hu
	for emacs-orgmode@gnu.org; Mon, 06 May 2013 07:08:26 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <lawrence.mitchell@ed.ac.uk>) id 1UZJHI-0008Jg-IN
	for emacs-orgmode@gnu.org; Mon, 06 May 2013 07:08:23 -0400
Received: from treacle.ucs.ed.ac.uk ([129.215.16.102]:32904)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <lawrence.mitchell@ed.ac.uk>) id 1UZJHI-0008Is-8t
	for emacs-orgmode@gnu.org; Mon, 06 May 2013 07:08:20 -0400
In-Reply-To: <87ip2xfd0x.fsf@gmail.com>
Content-Disposition: inline
List-Id: "General discussions about Org-mode." <emacs-orgmode.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/emacs-orgmode>,
	<mailto:emacs-orgmode-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/emacs-orgmode>
List-Post: <mailto:emacs-orgmode@gnu.org>
List-Help: <mailto:emacs-orgmode-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/emacs-orgmode>,
	<mailto:emacs-orgmode-request@gnu.org?subject=subscribe>
Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org
Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org
To: Nicolas Goaziou <n.goaziou@gmail.com>
Cc: Lawrence Mitchell <wence@gmx.li>, emacs-orgmode@gnu.org, Carsten Dominik <carsten.dominik@gmail.com>

[Reintroducing org mailing list CC]

On 05/05/2013 20:21, Nicolas Goaziou wrote:
> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
>>> I don't think there's much to do about that. Though, some tools could
>>> benefit from caching, like Lawrence did for
>>> `org-export-resolve-fuzzy-link'.
>>
>> Could you point out specific ones where it would make sense?  Maybe
>> someone would like to take this up as a task.
>
> It requires some careful benchmarking. Though, good candidates are tools
> searching for an object or element within the full parse tree. This
> includes:
>
>    - org-export-footnote-first-reference-p
>    - org-export-get-footnote-number
>    - org-export-get-category
>    - org-export-resolve-coderef
>    - org-export-resolve-radio-link
>    - org-export-get-loc
>    - org-export-get-ordinal

I wonder if it would be possible to store a copy of the parse tree in a 
form that is more amenable to log or constant time searches.

However, I note that my caching of the fuzzy link stuff brought the 
quadratic time export of copies of the org manual introduction down to 
linear (or close to) time.

Most of the problem now seems to be that for big documents, many 
functions are called a /lot/.  For example:

org-element--current-element takes (on my machine) 0.0003 seconds per 
call.  However, when exporting 128x the orgmanual introduction, it's 
called around 250000 times giving ~ 80 seconds total time (out of ~200 
total).

So it sort of feels like actually what is needed is microoptimisations 
of the bits of the export engine that are called the most.

Lawrence


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.