bug#43389: 28.0.50; Emacs memory leaks

[Eli Zaretskii <eliz@gnu.org>]

> From: Florian Weimer <fweimer@redhat.com>
> Cc: carlos@redhat.com,  dj@redhat.com,  43389@debbugs.gnu.org
> Date: Mon, 16 Nov 2020 21:42:39 +0100
> 
> * Eli Zaretskii:
> 
> > For some time (several months, I think) we have reports from Emacs
> > users that the memory footprints of their Emacs sessions sometimes
> > start growing very quickly, from several hundreds of MBytes to several
> > gigabytes in a day or even just few hours, and in some cases causing
> > the OOMK to kick in and kill the Emacs process.
> 
> The last time I saw this was a genuine memory leak in the Emacs C code.

That's always a possibility.  However, 2 aspects of these bug reports
seem to hint that there's more here than meets the eye:

 . the problem happens only to a small number of people, and it is
   hard to find an area in Emacs that would use memory in some special
   enough way to happen rarely

 . the Emacs sessions of the people who reported this would run for
   many days and even weeks on end with fairly normal memory footprint
   (around 500MB) that was very stable, and then suddenly begin
   growing by the minute to 10 or 20 times that

> There is an issue with reusing posix_memalign allocations.  On my system
> (running Emacs 27.1 as supplied by Fedora 32), I only see such
> allocations as the backing storage for the glib (sic) slab allocator.

(By "backing storage" you mean malloc calls that request large chunks
so that malloc obtains the memory from mmap?  Or do you mean something
else?)

Are the problems with posix_memalign also relevant to calls to
aligned_alloc?  Emacs calls the latter _a_lot_, see lisp_align_malloc.

> It gets exercised mostly when creating UI elements, as far as I can
> tell.

I guess your build uses GTK as the toolkit?

> There is actually a glibc patch floating around that fixes the aligned
> allocation problem, at some (hopefully limited) performance cost to
> aligned allocations.  We want to get it reviewed and integrated into
> upstream glibc.  If there is a working reproducer, we could run it
> against a patched glibc.

We don't have a reproducer, but several people said that the problem
happens to them regularly enough in their normal usage.  So I think we
can ask them to try a patches glibc and see if the problem goes away.

> The other issue we have is that thread counts has exceeded in recent
> times more than system memory, and glibc basically scales RSS overhead
> with thread count, not memory.  A use of libgomp suggests that many
> threads might indeed be spawned.  If their lifetimes overlap, it would
> not be unheard of to end up with some RSS overhead in the order of
> peak-usage-per-thread times 8 times the number of hardware threads
> supported by the system.  Setting MALLOC_ARENA_MAX to a small value
> counteracts that, so it's very simple to experiment with it if you have
> a working reproducer.

"Small value" being something like 2?

Emacs doesn't use libgomp, I think that comes from ImageMagick, and
most people who reported these problems use Emacs that wasn't built
with ImageMagick.  The only other source of threads in Emacs I know of
is GTK, but AFAIK it starts a small number of them, like 4.

In any case, experimenting with MALLOC_ARENA_MAX is easy, so I think
we should ask the people who experience this to try that.

Any other suggestions or thoughts?

Thanks.