Memory leaks

Memory leaks

[Daniel Clemente]

  Hi, today my Emacs --daemon from CVS started eating memory and ended up with 1 Gb resident and 2'5 Gb virtual, with only 180 small buffers open.

  I wanted to debug who was taking all this memory. Since I don't know how to find out this, I have started to collect more information in EmacsWiki: http://www.emacswiki.org/emacs/EmacsMemoryDebugging

  How do you debug memory leaks? Please add tips there so that they are useful to more people.

  Thanks

-- Daniel

Re: Memory leaks

[Chong Yidong]

Daniel Clemente <dcl441-bugs@yahoo.com> writes:

>   Hi, today my Emacs --daemon from CVS started eating memory and ended
>   up with 1 Gb resident and 2'5 Gb virtual, with only 180 small
>   buffers open.
>
>   I wanted to debug who was taking all this memory. Since I don't know
>   how to find out this, I have started to collect more information in
>   EmacsWiki: http://www.emacswiki.org/emacs/EmacsMemoryDebugging
>
>   How do you debug memory leaks? Please add tips there so that they
>   are useful to more people.

How quickly does this leak happen?  If you can make it leak quickly and
reproducibly, the easiest approach is probably to bisect.

Re: Memory leaks

[Daniel Clemente]

El ds, ago 08 2009 a les 20:57, Chong Yidong va escriure:
>>   Hi, today my Emacs --daemon from CVS started eating memory and ended
>> …
>
> How quickly does this leak happen?  If you can make it leak quickly and
> reproducibly, the easiest approach is probably to bisect.

  This was caused by an Emacs which didn't do garbage collection and warned: „Building Emacs overflowed pure space“, due to a recent change already fixed. The full story is now explained at http://www.emacswiki.org/emacs/EmacsMemoryDebugging#toc9

  I also added your tip about bisecting.

Memory leaks

[Uday S Reddy]

I am trying to track down some possible memory leaks in VM.  I have been 
assuming that, when a buffer gets killed,

- all its string memory is reclaimed,

- all the buffer-local variables are reclaimed, and

- all the overlays and intervals are reclaimed.

Are any of these assumptions invalid?  If so, what can I do about the 
lost memory?

Cheers,
Uday

Re: Memory leaks

[Andreas Schwab]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> I am trying to track down some possible memory leaks in VM.  I have been
> assuming that, when a buffer gets killed,
>
> - all its string memory is reclaimed,
>
> - all the buffer-local variables are reclaimed, and
>
> - all the overlays and intervals are reclaimed.
>
> Are any of these assumptions invalid?  If so, what can I do about the lost
> memory?

In general no memory is reclaimed until the next garbage collection (but
the buffer text is freed immediately).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leaks

[Uday S Reddy]

On 3/3/2011 5:55 PM, Andreas Schwab wrote:

> In general no memory is reclaimed until the next garbage collection (but
> the buffer text is freed immediately).

Yes, of course.  I guess they become "reclaimable" after a buffer is killed.

So, if I have memory leaking, it would seem that the only possibility is 
from global variables?

Cheers,
Uday

Re: Memory leaks

[Andreas Schwab]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> So, if I have memory leaking, it would seem that the only possibility is
> from global variables?

How do you know that there is some leak?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leaks

[Uday S Reddy]

On 3/3/2011 11:40 PM, Andreas Schwab wrote:

> How do you know that there is some leak?

First there was a suspicion, and now I have been running experiments.

Here are the profiles returned by (garbage-collect), pretty-printed, for 
the same Emacs session over time.  In each case, the Emacs session was 
essentially empty, with just a few stray text buffers .  The difference 
is only in what happened in between.

Initial:

(:conses (102590 . 13880)
  :syms (20328 . 54)
  :miscs (830 . 152)
  :chars 570345 :vector 329747 :floats (72 . 111)
  :intervals (348 . 189)
  :strings (21259 . 5705))

After one visit to my INBOX folder (22MB) and quitting:

(:conses (663630 . 605600)
  :syms (32257 . 2608)
  :miscs (15443 . 18022)
  :chars 1639236 :vector 454670 :floats (345 . 261)
  :intervals (503 . 622)
  :strings (99122 . 90256))

After a second visit to the same folder and quitting:

(:conses (1113362 . 334402)
  :syms (32264 . 2807)
  :miscs (15467 . 36265)
  :chars 1833568 :vector 456351 :floats (345 . 328)
  :intervals (604 . 849)
  :strings (138140 . 122806))

After several hours of usage:

(:conses (1558351 . 313757)
  :syms (93011 . 347)
  :miscs (140492 . 28025)
  :chars 9264181 :vector 2283330 :floats (347 . 638)
  :intervals (2771 . 4880)
  :strings (467731 . 34316))

At this point, the total memory occupied was about 400MB, and Emacs 
complained that "memory was exhausted; save buffers and quit" (or 
something to that effect).  I ran garbage-collect, killed all the stray 
buffers and left it around for a while (about 10-15 minutes), and the 
memory came down to 340MB.  (The memory usage figures are from the 
Windows XP Task Manger - "VM Size" - VM here referring to the virtual 
memory.)

After that, I could do a couple of more folder visits and then Emacs 
crashed in the middle of a folder visit.  Come to think of it, almost 
all the Emacs crashes I have experienced over the last year or so were 
probably related to the memory leak problem.  They all happened when I 
was visiting a folder.

Recalling from my memory, these kind of crashes have started happening 
with Emacs 22.3.  There were no such crashes before then.  I stuck to 
Emacs 22.2 for a long time, hoping that whatever was causing the crashes 
would get fixed in due course.  But with every new release, the same 
kind of crashes appeared.

Here is a crash report from yesterday produced by DrMinGW. 
Unfortunately, I was in the middle of a Lisp debugger session and the 
crash happened while the debugger was trying print a large data 
structure.  But this does look like a memory problem to my untrained eye.

Cheers,
Uday


---------------


emacs.exe caused an Access Violation at location 77c47412 in module 
msvcrt.dll Writing to location 18874000.

Registers:
eax=00829240 ebx=02bb2802 ecx=00000000 edx=00000001 esi=008292bf 
edi=18874000
eip=77c47412 esp=00829290 ebp=00829298 iopl=0         nv up ei ng nz ac 
pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000 
efl=00000293

Call stack:
77C47412  msvcrt.dll:77C47412  memmove
0110AD0F  emacs.exe:0110AD0F  printchar  print.c:323

0110FC03  emacs.exe:0110FC03  print_object  print.c:1645

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0110EAE2  emacs.exe:0110EAE2  print  print.c:1305

0110CF92  emacs.exe:0110CF92  Fprin1  print.c:751

01022B04  emacs.exe:01022B04  Ffuncall  eval.c:3034

01114C11  emacs.exe:01114C11  Fbyte_code  bytecode.c:680

010232C3  emacs.exe:010232C3  funcall_lambda  eval.c:3220

01022D03  emacs.exe:01022D03  Ffuncall  eval.c:3077

01114C11  emacs.exe:01114C11  Fbyte_code  bytecode.c:680

01021B46  emacs.exe:01021B46  Feval  eval.c:2356

0101ED68  emacs.exe:0101ED68  Fprogn  eval.c:415

01102A41  emacs.exe:01102A41  Fsave_window_excursion  window.c:6575

0111579A  emacs.exe:0111579A  Fbyte_code  bytecode.c:841

010232C3  emacs.exe:010232C3  funcall_lambda  eval.c:3220

01022D03  emacs.exe:01022D03  Ffuncall  eval.c:3077

010221D6  emacs.exe:010221D6  Fapply  eval.c:2506

010225FD  emacs.exe:010225FD  apply1  eval.c:2780

0101EB72  emacs.exe:0101EB72  call_debugger  eval.c:272

01022E12  emacs.exe:01022E12  Ffuncall  eval.c:3102

01114C11  emacs.exe:01114C11  Fbyte_code  bytecode.c:680

01021B46  emacs.exe:01021B46  Feval  eval.c:2356

01020165  emacs.exe:01020165  internal_catch  eval.c:1228

0111580E  emacs.exe:0111580E  Fbyte_code  bytecode.c:855

010232C3  emacs.exe:010232C3  funcall_lambda  eval.c:3220

01022D03  emacs.exe:01022D03  Ffuncall  eval.c:3077

010221D6  emacs.exe:010221D6  Fapply  eval.c:2506

010225FD  emacs.exe:010225FD  apply1  eval.c:2780

01117B84  emacs.exe:01117B84  Fcall_interactively  callint.c:396

01022B2E  emacs.exe:01022B2E  Ffuncall  eval.c:3037

010226CC  emacs.exe:010226CC  call3  eval.c:2857

01014D52  emacs.exe:01014D52  Fcommand_execute  keyboard.c:10562

010150C3  emacs.exe:010150C3  Fexecute_extended_command  keyboard.c:10675

01022AE3  emacs.exe:01022AE3  Ffuncall  eval.c:3031

0111932A  emacs.exe:0111932A  Fcall_interactively  callint.c:869

01022B2E  emacs.exe:01022B2E  Ffuncall  eval.c:3037

010226CC  emacs.exe:010226CC  call3  eval.c:2857

01014D52  emacs.exe:01014D52  Fcommand_execute  keyboard.c:10562

010076E0  emacs.exe:010076E0  command_loop_1  keyboard.c:1914

01020614  emacs.exe:01020614  internal_condition_case  eval.c:1492

01005FE3  emacs.exe:01005FE3  command_loop_2  keyboard.c:1362

01020165  emacs.exe:01020165  internal_catch  eval.c:1228

01005F9C  emacs.exe:01005F9C  command_loop  keyboard.c:1342

0100588E  emacs.exe:0100588E  recursive_edit_1  keyboard.c:956

010059F7  emacs.exe:010059F7  Frecursive_edit  keyboard.c:1019

010028F5  emacs.exe:010028F5  main  emacs.c:1835

010010DB  emacs.exe:010010DB
01001178  emacs.exe:01001178
010036D0  emacs.exe:010036D0  _start  unexw32.c:131

7C817077  kernel32.dll:7C817077  RegisterWaitForInputIdle

Re: Memory leaks

[Eli Zaretskii]

> From: Uday S Reddy <u.s.reddy@cs.bham.ac.uk>
> Date: Fri, 04 Mar 2011 11:39:05 +0000
> Cc: u.s.reddy@cs.bham.ac.uk
> 
> Initial:
> 
> (:conses (102590 . 13880)
>   :syms (20328 . 54)
>   :miscs (830 . 152)
>   :chars 570345 :vector 329747 :floats (72 . 111)
>   :intervals (348 . 189)
>   :strings (21259 . 5705))
> 
> After one visit to my INBOX folder (22MB) and quitting:
> 
> (:conses (663630 . 605600)
>   :syms (32257 . 2608)
>   :miscs (15443 . 18022)
>   :chars 1639236 :vector 454670 :floats (345 . 261)
>   :intervals (503 . 622)
>   :strings (99122 . 90256))
> 
> After a second visit to the same folder and quitting:
> 
> (:conses (1113362 . 334402)
>   :syms (32264 . 2807)
>   :miscs (15467 . 36265)
>   :chars 1833568 :vector 456351 :floats (345 . 328)
>   :intervals (604 . 849)
>   :strings (138140 . 122806))

Is this with Rmail or with some other mail agent?

With Rmail, I cannot reproduce this with the current trunk.  Here are
my results of garbage-collect, the first one is the initial one, the
others are pairs, where the first one is after visiting my 15MB INBOX,
the second one after killing the RMAIL buffer:

((49547 . 8089) (14061 . 0) (21 . 113) 63254 292573 (44 . 78) (322 . 66) (3574 . 2033))

((69953 . 4823) (16430 . 0) (2030 . 3583) 219252 322528 (49 . 73) (373 . 128) (10009 . 4103))

((67776 . 6125) (16430 . 0) (33 . 246) 217261 316518 (49 . 73) (365 . 136) (10006 . 4106))

((70073 . 6199) (16434 . 0) (2032 . 3089) 219400 322528 (49 . 73) (388 . 164) (10016 . 4096))

((67867 . 6905) (16434 . 0) (35 . 340) 217317 316510 (49 . 73) (379 . 137) (10010 . 4102))

((70162 . 6779) (16438 . 0) (2034 . 3007) 219418 322528 (49 . 73) (400 . 167) (10018 . 4094))

((67956 . 7360) (16438 . 0) (37 . 342) 217335 316510 (49 . 73) (391 . 176) (10012 . 4100))

You can see that the counters go up very slowly, much slower than what
you report.

In fact, the initial report in my cases shows counters that are
several times smaller than in your case.  Can you explain this
discrepancy?

I see similar numbers in Emacs 23.3, so it's not like the trunk is
very different.

Re: Memory leaks

[Uday S Reddy]

Eli Zaretskii writes:

> Is this with Rmail or with some other mail agent?

Hi Eli, my figures for VM, the mail agent that I maintain.  It is much
more of a feature-laden system.  So, it probably uses more memory.
But the problem I am after is the one that the memory is leaking, it
is not getting reused.

I don't know yet whether it is a VM problem or an Emacs problem.  I am
still trying to understand what is going on.

Any tips would be great!

Cheers,
Uday

Re: Memory leaks

[Uday S Reddy]

On 3/4/2011 11:39 AM, Uday S Reddy wrote:

> After several hours of usage:
>
> (:conses (1558351 . 313757)
> :syms (93011 . 347)
> :miscs (140492 . 28025)
> :chars 9264181 :vector 2283330 :floats (347 . 638)
> :intervals (2771 . 4880)
> :strings (467731 . 34316))

One of the things I have decided to check is what I am doing with 
vectors because more than 2M vector slots were being locked up in an 
empty Emacs session.

A widespread use of vectors in VM is for obarrays.  The code might look 
something like this:

    (let ((obarray (make-vector 67 0)))
       ... intern lots of symbols in obarray
       (setq buffer-local-var obarray))

I have begun to wonder if obarray gets freed after buffer-local-var 
stops pointing to it, say through another setq to buffer-local-var.

Am I supposed to unintern the symbols to free up their memory?

Cheers,
Uday

Re: Memory leaks

[Andreas Schwab]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> I have begun to wonder if obarray gets freed after buffer-local-var stops
> pointing to it, say through another setq to buffer-local-var.

Obarrays are ordinary vectors, that get garbage collected like any other
object.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leaks

[David Kastrup]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:
>
>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>> pointing to it, say through another setq to buffer-local-var.
>
> Obarrays are ordinary vectors, that get garbage collected like any other
> object.

Unfortunately, symbols are daisy-chained from their obarray bucket, so
even unused symbols can't be collected until all their successors in the
hidden hash bucket chain have become unused as well.

-- 
David Kastrup

Re: Memory leaks

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:
>>
>>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>>> pointing to it, say through another setq to buffer-local-var.
>>
>> Obarrays are ordinary vectors, that get garbage collected like any other
>> object.
>
> Unfortunately, symbols are daisy-chained from their obarray bucket, so
> even unused symbols can't be collected until all their successors in
> the hidden hash bucket chain have become unused as well.

Their predecessors I mean.

-- 
David Kastrup

Re: Memory leaks

[Uday S Reddy]

On 3/7/2011 8:33 PM, Andreas Schwab wrote:
> Uday S Reddy<u.s.reddy@cs.bham.ac.uk>  writes:
>
>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>> pointing to it, say through another setq to buffer-local-var.
>
> Obarrays are ordinary vectors, that get garbage collected like any other
> object.

I didn't think the vectors themselves would be in trouble, but rather 
what is behind the vectors: the buckets, the symbols and so on.  Even 
after the vectors are gone, there could be references to the symbols in 
other places, could there not?  So, it is not entirely clear when all 
that memory could be collected?

Cheers,
Uday

Re: Memory leaks

[Stefan Monnier]

>>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>>> pointing to it, say through another setq to buffer-local-var.
>> Obarrays are ordinary vectors, that get garbage collected like any other
>> object.
> I didn't think the vectors themselves would be in trouble, but rather what
> is behind the vectors: the buckets, the symbols and so on.  Even after the
> vectors are gone, there could be references to the symbols in other places,
> could there not?  So, it is not entirely clear when all that memory could
> be collected?

But then you'd have lots of symbols, not lots of vectors.


        Stefan

Re: Memory leaks

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>>>> pointing to it, say through another setq to buffer-local-var.
>>> Obarrays are ordinary vectors, that get garbage collected like any other
>>> object.
>> I didn't think the vectors themselves would be in trouble, but rather what
>> is behind the vectors: the buckets, the symbols and so on.  Even after the
>> vectors are gone, there could be references to the symbols in other places,
>> could there not?  So, it is not entirely clear when all that memory could
>> be collected?
>
> But then you'd have lots of symbols, not lots of vectors.

Unless those symbols have lots of vectors as some of their values.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leaks

[Richard Stallman]

    Unfortunately, symbols are daisy-chained from their obarray bucket, so
    even unused symbols can't be collected until all their successors in the
    hidden hash bucket chain have become unused as well.

It would not be hard to make GC know how to GC symbols and delete them
from the obarray.  This is a traditional Lisp feature that I left out
to make Emacs smaller.

However, it won't do much good unless you create a lot of symbols that
you don't need for long.  Is that happening?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: Memory leaks

[Richard Stallman]

Is it possible to write code to record which vectors were consed by
the operation suspected of having a memory leak?  Then a function to
call from GDB to test whether a vector are still alive.

After you find one, you could do a GC with a conditional breakpoint
to stop when that vector is marked.  Then you would see exactly
what is keeping it alive.  If you do this for each vector that was
consed during the operation in question, you will eventually find
one that was expected to be GC'd, and find the bug.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: Memory leaks

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> Is it possible to write code to record which vectors were consed by
> the operation suspected of having a memory leak?  Then a function to
> call from GDB to test whether a vector are still alive.

It should be possible to record them in a key-weak hashtable.  If they
don't leave after explicitly calling the garbage collector, you've got a
leak.

-- 
David Kastrup