Org element cache in indirect buffers

Org element cache in indirect buffers

[Sebastian Wålinder]

Hello!

It appears that indirect buffers duplicate the entire org element cache from the base buffer whenever created.

Running `memory-report` indicates that each indirect buffer has the same memory blueprint.

This becomes a massive issue with large org files.

My in my 160 000 line org file, the org element cache takes 100 mb, and so each new indirect buffer consumes an additional 100 mbs of memory.

I also wonder if this means the caching is performed by each indirect buffer independently, and doesn't carry over.

Thoughts?

Thanks,
Sebastian

Re: Org element cache in indirect buffers

[Ihor Radchenko]

Sebastian Wålinder <s.walinder@gmail.com> writes:

> It appears that indirect buffers duplicate the entire org element cache from the base buffer whenever created.

Not exactly. Only link to the cache root is copied and some cache
elements may not be garbage-collected.

> Running `memory-report` indicates that each indirect buffer has the same memory blueprint.

> This becomes a massive issue with large org files.
>
> My in my 160 000 line org file, the org element cache takes 100 mb, and so each new indirect buffer consumes an additional 100 mbs of memory.

This has to be a bug in `memory-report'. Most of the cache elements will
be referenced by link and thus there should be double-counting going on somewhere.

> I also wonder if this means the caching is performed by each indirect buffer independently, and doesn't carry over.

Caching is using base buffer actually.
I guess we can explicitly set reference to the cache root to nil in
indirect buffers.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org element cache in indirect buffers

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>> I also wonder if this means the caching is performed by each indirect buffer independently, and doesn't carry over.
>
> Caching is using base buffer actually.
> I guess we can explicitly set reference to the cache root to nil in
> indirect buffers.

Done, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=2f7b35ac8
But I would not expect much actual gain from this. Maybe in some edge cases.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org element cache in indirect buffers

[Ihor Radchenko]

[Adding Org ML back to CC]

Sebastian <s.walinder@gmail.com> writes:

> Thank you so much for the detailed explanation!
>
> Maybe memory-report is counting what's at the end of the reference, and
> your change fixes it.

memory-report is counting buffer data separately, even when buffer-local
variables reference to the same data objects.
I just reported this upstream: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=64050

My change simply clears the references to AVL-tree roots that are copied
over when Emacs clones the buffer. The edge cases I was talking about
are when the main AVL-tree changes its root. That would make indirect
buffer actually keep reference to obsolete root and the rest of the
obsolete AVL-tree, preventing it from being garbage-collected.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>