bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

[Trevor Bentley <trevor@trevorbentley.com>]

Eli Zaretskii <eliz@gnu.org> writes: 
>> Some interesting observations: 
>>  - (garbage-collect) takes forever, like on the order of 5-10 
>>  minutes, with one CPU core pegged to 100% and emacs frozen. 
> 
> Is this with the default values of gc-cons-threshold and 
> gc-cons-percentage? 

Yes, and they're both printed in the logs: threshold 800000, 
percentage 0.1.
 
>>  - The leaking stops for a while after (garbage-collect).  It 
>>  was  leaking 1MB per second for this last log, and stopped 
>>  growing  after the garbage collection. 
> 
> Now, what happens in that session once per second (in an 
> otherwise idle Emacs, I presume?) to cause such memory 
> consumption?  Some timers?  If you run with a breakpoint in 
> malloc that just shows the backtrace and continues, do you see 
> what could consume 1MB every second? 

Not an idle emacs at all, in this case.  I have seen the memory 
growth in an idle emacs, but the only one I can reproduce it on is 
the emacs-slack one, which is connected to a corporate Slack 
account.  Tons of short messages streaming in over the network and 
being displayed in rotating buffers, with images mixed in.  It's a 
big 'ol "web 2.0" API... it can easily pass 1MB/s of bloated JSON 
messages through.  This is one _very active_ emacs.

The original strace logs and valgrind output I posted before 
showed a random assortment of calls from gnutls, imagemagick, and 
lisp strings, with lisp strings dominating the malloc calls 
(enlarge_buffer_text, mostly).
 
>> Is there any way to get the same info without actually reaping 
>> dead references? 
> 
> What do you mean by "reaping dead references" here? 
> 
>> It could be that there really were 4.3GB of dead references. 
> 
> Not sure I understand what are you trying to establish here. 
>

GC is running through a list of active allocations and freeing the 
ones with no remaining references, right?  Presumably, if a lot of 
active malloc() allocations are no longer refernced, and 
(garbage-collect) calls free() on a bunch of blocks.  I'm 
wondering how to figure out how much memory a call to 
(garbage-collect) has actually freed.  Possibly a sort of "dry 
run" where it performs the GC algorithm, but doesn't release any 
memory.

(I'm very much assuming how emacs memory management works.  Please 
corect me if I'm wrong.)
 
> There's only one garbage-collect, it is called for _any_ GC. 
> 
> What do you mean by "during normal use" in this sentence: 
> 
>   I certainly don't notice 5-10 minute long pauses during normal 
>   use, though "gcs-done" is incrementing. 
> 
> How is what you did here, where GC took several minutes, 
> different from "normal usage"?

In this log, I am explicitly executing "(garbage-collect)", and it 
takes 10 minutes, during which the UI is unresponsive and 
sometimes even turns grey when the window stops redrawing.

By "normal use", I mean that I use this emacs instance on-and-off 
all day long.  I would notice if it were freezing for minutes at a 
time, and it definitely is not.

As far as I understand, garbage collection is supposed to happen 
automatically during idle.  I would certainly notice if it locked 
up the whole instance for 10 minutes from an idle GC.  I think 
this means the automatic garbage collection is either not 
happening, or running on a different thread, or being interrupted, 
or simply works differently.  I have no idea, hence asking you :)

The confusing part is that "gcs-done" increments a lot between my 
manual (garbage-collect) calls.  It looks like it does about 500 
per day.  There is no way emacs freezes and pegs a CPU core to max 
500 times per day, but it does exactly that every time I manually 
execute garbage-collect. 

Side note: it inflated to 7670MB overnight.  I'm running 
(garbage-collect) as I type this, but it has been churning for 30 
minutes with the UI frozen, and still isn't done.  I'm going to 
give up and kill it if it doesn't finish soon, as I kind of need 
that 8GB back.
 
-Trevor