Re: Question about (excessive?) ram usage when many overlays (with large vscroll)

[Eli Zaretskii <eliz@gnu.org>]

> Date: Tue, 26 Apr 2022 15:33:08 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: dalanicolai <dalanicolai@gmail.com>
> > Date: Tue, 26 Apr 2022 14:21:07 +0200
> > 
> > I have a question about ram usage when using overlays.
> > 
> > So I have created `image-roll.el` for displaying documents/books (see here
> > <https://lists.gnu.org/archive/html/emacs-devel/2022-04/msg00975.html>).
> > However, I have just noticed that it uses a large amount of RAM when
> > viewing (or
> > trying to) pages in the back of 'large' books. But even if RAM usage still
> > looks
> > perfectly fine, Emacs crashes when trying to scroll to higher page numbers.
> > 
> > I have looked a little into it, and have found that it is a consequence of
> > using
> > large overlays. There is no problem when creating a buffer containing many
> > overlays, however, when trying to scroll to some overlay at the end of the
> > buffer,
> > Emacs will use huge amounts of RAM.
> > 
> > So I am wondering if this is 'necessary' behavior; and then why is it
> > necessary?
> > The issue occurs even when displaying the same 'empty' svg-image on each
> > overlay, but it occurs also when simply displaying some 'specified space' on
> > each overlay.
> 
> I didn't yet try to reproduce this, but could you perhaps run this
> test under GDB, and when Emacs crashes, show the C-level backtrace?
> This could give good hints about the possible reason(s).

What I see here is that the memory footprint indeed goes up quite
quickly, but then (not sure exactly what triggers that in my case), it
gets reset back to almost the original small value.

If this doesn't happen for you, then I guess your code somehow
triggers bad behavior of glibc's malloc, forcing it not to release
memory back to the OS, due to the particular pattern of allocations
and deallocations?  Or maybe something else is at work here and I just
got lucky?

Memory is allocated for dealing with overlays, I think, because
redisplay needs to have the overlays centered around the position the
text it is rendering, and moving around many hundreds of overlays
needs memory for the move.

But that's a guess.  If someone can find out why we allocate large
amounts of memory in this scenario, that could perhaps help us
understand better what's going on here.