Scanning for duplicate IDs of whole buffer when exporting a sub-heading

Scanning for duplicate IDs of whole buffer when exporting a sub-heading

[Karl Voit]

Hi!

I do have this behavior which basically prevents me from exporting
from Org-mode. When I invoke export (LaTeX, HTML, ...) of a
sub-heading within a large file, Org-mode seems to scan *all* of my
agenda files (with approximately half a Million lines) at least for
duplicate IDs.

This takes very long on my i5 machine, roughly ten minutes. During
this scan, I can't use Emacs any more because it's blocking of
course.

How can I prevent this behavior?

The current sub-heading (with no further sub-headings) does not even
have any drawer.

Thanks for any hint!

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: Scanning for duplicate IDs of whole buffer when exporting a sub-heading

[Nicolas Goaziou]

Hello,

Karl Voit <devnull@Karl-Voit.at> writes:

> I do have this behavior which basically prevents me from exporting
> from Org-mode. When I invoke export (LaTeX, HTML, ...) of a
> sub-heading within a large file, Org-mode seems to scan *all* of my
> agenda files (with approximately half a Million lines) at least for
> duplicate IDs.

Does your large file contain some [[id:...]] link? Does the sub-heading
contains any?

> This takes very long on my i5 machine, roughly ten minutes. During
> this scan, I can't use Emacs any more because it's blocking of
> course.

Even when using asynchronous export?

Regards,

-- 
Nicolas Goaziou

Re: Scanning for duplicate IDs of whole buffer when exporting a sub-heading

[Samuel Wales]

reported before. maybe nicolas fixed it in master but not maint.

Re: Scanning for duplicate IDs of whole buffer when exporting a sub-heading

[Karl Voit]

* Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Hello,

Hi Nicolas!

> Karl Voit <devnull@Karl-Voit.at> writes:
>
>> I do have this behavior which basically prevents me from exporting
>> from Org-mode. When I invoke export (LaTeX, HTML, ...) of a
>> sub-heading within a large file, Org-mode seems to scan *all* of my
>> agenda files (with approximately half a Million lines) at least for
>> duplicate IDs.
>
> Does your large file contain some [[id:...]] link? 

Yes.

> Does the sub-heading contains any?

No.

>> This takes very long on my i5 machine, roughly ten minutes. During
>> this scan, I can't use Emacs any more because it's blocking of
>> course.
>
> Even when using asynchronous export?

Yes.

I am on maint, 75ee067410abf0ebab77b1b05d7dcfe2ddf53f4d.

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: Scanning for duplicate IDs of whole buffer when exporting a sub-heading

[Nicolas Goaziou]

Hello,

Karl Voit <devnull@Karl-Voit.at> writes:

>> Does your large file contain some [[id:...]] link? 
>
> Yes.
>
>> Does the sub-heading contains any?
>
> No.
>
>>> This takes very long on my i5 machine, roughly ten minutes. During
>>> this scan, I can't use Emacs any more because it's blocking of
>>> course.
>>
>> Even when using asynchronous export?
>
> Yes.
>
> I am on maint, 75ee067410abf0ebab77b1b05d7dcfe2ddf53f4d.

As Samuel pointed out, this was fixed in master some time ago. 

Meanwhile you could, as a hack, remove all [[id:...]] links from your
document with a function in `org-export-before-processing-hook'.

Regards,

-- 
Nicolas Goaziou

Re: Scanning for duplicate IDs of whole buffer when exporting a sub-heading

[Karl Voit]

* Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Hello,

Hi Nicolas!

> Karl Voit <devnull@Karl-Voit.at> writes:
>
>>> Does your large file contain some [[id:...]] link? 
>>
>> Yes.
>> I am on maint, 75ee067410abf0ebab77b1b05d7dcfe2ddf53f4d.

I should think of automatically adding my current emacs + Org-mode
version in my email/posting footer :-)

> As Samuel pointed out, this was fixed in master some time ago. 

I see.

What is the policy of merging changes from master to maint? Does
this happen only for major issues? Only before releases?

Thanks for clarification!

> Meanwhile you could, as a hack, remove all [[id:...]] links from your
> document with a function in `org-export-before-processing-hook'.

Sorry, I'm a bit confused here. Does this mean that id-links are
replaced by something new or do you assume that the id-links in my
document are not important (because it ought to be only a small
file for a presentation)?

If it's the latter: my Org-mode file is quite large and contains
hundreds of id-links I want to keep. The sub-heading of the
presentation I want to export is only a tiny fraction of the file.

My whole digital life is managed in a dozen Org-mode files with a
quarter million lines and approximately ninety percent is within
five files. Therefore I love narrowing down to sub-headings for
editing and exporting.

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: Scanning for duplicate IDs of whole buffer when exporting a sub-heading

[Nicolas Goaziou]

Hello,

Karl Voit <devnull@Karl-Voit.at> writes:

> What is the policy of merging changes from master to maint? Does
> this happen only for major issues? Only before releases?

Usually maint contains either very important or trivial fixes. To put it
differently, fixes that might not go in maint are invasive changes that
solve non-critical issues. I believe this one fell in the latter
category.

However, as you can tell, there is usually no clear cut.

> Sorry, I'm a bit confused here. Does this mean that id-links are
> replaced by something new or do you assume that the id-links in my
> document are not important (because it ought to be only a small
> file for a presentation)?
>
> If it's the latter: my Org-mode file is quite large and contains
> hundreds of id-links I want to keep. The sub-heading of the
> presentation I want to export is only a tiny fraction of the file.
>
> My whole digital life is managed in a dozen Org-mode files with a
> quarter million lines and approximately ninety percent is within
> five files. Therefore I love narrowing down to sub-headings for
> editing and exporting.

Export process happens in a copy of your documents. You can safely
remove id links there (since they get in the way ATM) using hooks.

Regards,

-- 
Nicolas Goaziou