bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

[Carlos O'Donell <carlos@redhat.com>]

On 11/24/20 11:07 AM, Eli Zaretskii wrote:
> Look at the large chunks in the tail of this.  Together, they do
> account for ~2GB.
> 
> Carlos, are these chunks in use (i.e. allocated and not freed), or are
> they the free chunks that are available for allocation, but not
> released to the OS?  If the former, then it sounds like this session
> does have around 2GB of allocated heap data, so either there's some
> allocated memory we don't account for, or there is indeed a memory
> leak in Emacs.  If these are the free chunks, then the way glibc
> manages free'd memory is indeed an issue.

These chunks are all free and mapped for use by the algorithm to satisfy
a request by the application.

Looking at the last malloc_info (annotated):
https://trevorbentley.com/emacs_malloc_info.log
===============================================
;; malloc-info
(malloc-info)
<malloc version="1">
<heap nr="0">
<sizes>
</sizes>
<total type="fast" count="0" size="0"/>

=> No fast bins.

<total type="rest" count="1" size="112688"/>

=> 1 unused bin.

=> In total we have only 112KiB in 1 unused chunk free'd on the stack.
=> The rest of the stack is in use by the application.
=> It looks like the application usage goes down to zero and then up again?

<system type="current" size="4243079168"/>

=> Currently at 4.2GiB in arena 0 (kernel assigned heap).
=> The application is using that sbrk'd memory.

<system type="max" size="4243079168"/>
<aspace type="total" size="4243079168"/>
<aspace type="mprotect" size="4243079168"/>

=> This indicates *real* API usage of 4.2GiB.

</heap>
<heap nr="1">

=> This is arena 1, which is a thread heap, and uses mmap to create heaps.

<sizes>
  <size from="17" to="32" total="32" count="1"/>
  <size from="33" to="48" total="240" count="5"/>
  <size from="49" to="64" total="256" count="4"/>
  <size from="65" to="80" total="160" count="2"/>
  <size from="97" to="112" total="224" count="2"/>
  <size from="33" to="33" total="231" count="7"/>
  <size from="49" to="49" total="294" count="6"/>
  <size from="65" to="65" total="390" count="6"/>
  <size from="81" to="81" total="162" count="2"/>
  <size from="97" to="97" total="97" count="1"/>
  <size from="129" to="129" total="516" count="4"/>
  <size from="161" to="161" total="644" count="4"/>
  <size from="209" to="209" total="1254" count="6"/>
  <size from="241" to="241" total="241" count="1"/>
  <size from="257" to="257" total="257" count="1"/>
  <size from="305" to="305" total="610" count="2"/>
  <size from="32209" to="32209" total="32209" count="1"/>
  <size from="3982129" to="8059889" total="28065174" count="6"/>
  <unsorted from="209" to="4020593" total="4047069" count="13"/>
</sizes>
<total type="fast" count="14" size="912"/>
<total type="rest" count="61" size="42357420"/>

=> Pretty small, 912 bytes in fastbins, and 42MiB in cached chunks.

<system type="current" size="42426368"/>
<system type="max" size="42426368"/>
<aspace type="total" size="42426368"/>
<aspace type="mprotect" size="42426368"/>
<aspace type="subheaps" size="1"/>
</heap>
<total type="fast" count="14" size="912"/>
<total type="rest" count="62" size="42470108"/>
<total type="mmap" count="9" size="208683008"/>
<system type="current" size="4285505536"/>
<system type="max" size="4285505536"/>
<aspace type="total" size="4285505536"/>
<aspace type="mprotect" size="4285505536"/>
</malloc>
===============================================

This shows the application is USING memory on the main system heap.

It might not be "leaked" memory since the application might be using it.

You want visibility into what is USING that memory.

With glibc-malloc-trace-utils you can try to do that with:

LD_PRELOAD=libmtrace.so \
MTRACE_CTL_FILE=/home/user/app.mtr \
MTRACE_CTL_BACKTRACE=1 \
./app

This will use libgcc's unwinder to get a copy of the malloc caller
address and then we'll have to decode that based on a /proc/self/maps.

Next steps:
- Get a glibc-malloc-trace-utils trace of the application ratcheting.
- Get a copy of /proc/$PID/maps for the application (shorter version of smaps).

Then we might be able to correlate where all the kernel heap data went?

-- 
Cheers,
Carlos.