On Fri, Dec 17, 2021 at 11:34 PM Ihor Radchenko <yantar92@gmail.com> wrote:
> There's a setup included in there; I forgot about that. If you don't mind
> trying it again, this would download everything needed:

Thanks! I was able to follow your steps this time.

I really appreciate making this extra effort to help me out. Thanks!
 
> Sorry, but I am getting similar backtrace in another function now:

Sigh... Should be fixed now. I did not expect that arguments to cache
functions could ever be markers.

After the fix, I was able to see the warning you were talking about in
the earlier email. Your test case revealed an important omission in the
cache logic when a huge number of edits are stacked together into
asynchronous cache queue. Thanks you for providing the excellent
reproducer!

No problem! The ox-hugo tests end up indirectly testing a lot of Org features :D
 
On latest main, your example file is exported without any warnings.

I just verified that. Also, I did not even need to add `(org-element-cache-reset)' in `org-hugo--get-pre-processed-buffer' after the `(insert ..)' in the temp buffer.

Is that expected?

Also, I noticed that now the exports are much more snappier than before!

Also, I noticed that the large number of edits makes the cache use a lot
of memory adding load onto the Emacs garbage collector. May I know what
exactly ox-hugo does that triggers that many edits? Exporting your
example files took approximately 30sec on my system (both with and
without cache). I believe that the speed can be improved if cache is
used more optimally.

That command exports 81 subtrees (as of now) from ox-hugo-manual.org to independent Markdown files. 

That said, I think that `org-hugo--get-pre-processed-buffer' [1] is the performance bottleneck. It was added so that Org subtree exports can work even if one subtree has an internal link to a different subtree in the same file.

The pre processing logic slurps up the AST of the whole Org buffer, does org-element-copy on all the link org-elements to point to a place outside the exported subtree, fixes that link so that the export works, and copies the modified link element back on the original element. And then the whole Org temp buffer is overwritten with the modified AST.
And then.. the subtree export happens from there.

This is the only way possible as far as I know to support cross-subtree links. Credit for this implementation goes to Folkert van der Beek[2].

Let me know if you have any suggestions to improve the performance. I think that the org-element-copy and writing the huge AST to temp Org buffer is keeping the GC busy (the ox-hugo-manual.org is about 3700 lines right now).

[1]: https://github.com/kaushalmodi/ox-hugo/blob/1ea466d33f76eaf2567593fa612b82c22ef86d3d/ox-hugo.el#L4033-L4151

[2]: https://github.com/kaushalmodi/ox-hugo/pull/280