Memory leaks

Memory leaks

[Daniel Clemente]

  Hi, today my Emacs --daemon from CVS started eating memory and ended up with 1 Gb resident and 2'5 Gb virtual, with only 180 small buffers open.

  I wanted to debug who was taking all this memory. Since I don't know how to find out this, I have started to collect more information in EmacsWiki: http://www.emacswiki.org/emacs/EmacsMemoryDebugging

  How do you debug memory leaks? Please add tips there so that they are useful to more people.

  Thanks

-- Daniel

Re: Memory leaks

[Chong Yidong]

Daniel Clemente <dcl441-bugs@yahoo.com> writes:

>   Hi, today my Emacs --daemon from CVS started eating memory and ended
>   up with 1 Gb resident and 2'5 Gb virtual, with only 180 small
>   buffers open.
>
>   I wanted to debug who was taking all this memory. Since I don't know
>   how to find out this, I have started to collect more information in
>   EmacsWiki: http://www.emacswiki.org/emacs/EmacsMemoryDebugging
>
>   How do you debug memory leaks? Please add tips there so that they
>   are useful to more people.

How quickly does this leak happen?  If you can make it leak quickly and
reproducibly, the easiest approach is probably to bisect.

Re: Memory leaks

[Daniel Clemente]

El ds, ago 08 2009 a les 20:57, Chong Yidong va escriure:
>>   Hi, today my Emacs --daemon from CVS started eating memory and ended
>> …
>
> How quickly does this leak happen?  If you can make it leak quickly and
> reproducibly, the easiest approach is probably to bisect.

  This was caused by an Emacs which didn't do garbage collection and warned: „Building Emacs overflowed pure space“, due to a recent change already fixed. The full story is now explained at http://www.emacswiki.org/emacs/EmacsMemoryDebugging#toc9

  I also added your tip about bisecting.

Memory leaks

[Uday S Reddy]

I am trying to track down some possible memory leaks in VM.  I have been 
assuming that, when a buffer gets killed,

- all its string memory is reclaimed,

- all the buffer-local variables are reclaimed, and

- all the overlays and intervals are reclaimed.

Are any of these assumptions invalid?  If so, what can I do about the 
lost memory?

Cheers,
Uday

Re: Memory leaks

[Andreas Schwab]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> I am trying to track down some possible memory leaks in VM.  I have been
> assuming that, when a buffer gets killed,
>
> - all its string memory is reclaimed,
>
> - all the buffer-local variables are reclaimed, and
>
> - all the overlays and intervals are reclaimed.
>
> Are any of these assumptions invalid?  If so, what can I do about the lost
> memory?

In general no memory is reclaimed until the next garbage collection (but
the buffer text is freed immediately).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leaks

[Uday S Reddy]

On 3/3/2011 5:55 PM, Andreas Schwab wrote:

> In general no memory is reclaimed until the next garbage collection (but
> the buffer text is freed immediately).

Yes, of course.  I guess they become "reclaimable" after a buffer is killed.

So, if I have memory leaking, it would seem that the only possibility is 
from global variables?

Cheers,
Uday

Re: Memory leaks

[Andreas Schwab]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> So, if I have memory leaking, it would seem that the only possibility is
> from global variables?

How do you know that there is some leak?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leaks

[Uday S Reddy]

On 3/3/2011 11:40 PM, Andreas Schwab wrote:

> How do you know that there is some leak?

First there was a suspicion, and now I have been running experiments.

Here are the profiles returned by (garbage-collect), pretty-printed, for 
the same Emacs session over time.  In each case, the Emacs session was 
essentially empty, with just a few stray text buffers .  The difference 
is only in what happened in between.

Initial:

(:conses (102590 . 13880)
  :syms (20328 . 54)
  :miscs (830 . 152)
  :chars 570345 :vector 329747 :floats (72 . 111)
  :intervals (348 . 189)
  :strings (21259 . 5705))

After one visit to my INBOX folder (22MB) and quitting:

(:conses (663630 . 605600)
  :syms (32257 . 2608)
  :miscs (15443 . 18022)
  :chars 1639236 :vector 454670 :floats (345 . 261)
  :intervals (503 . 622)
  :strings (99122 . 90256))

After a second visit to the same folder and quitting:

(:conses (1113362 . 334402)
  :syms (32264 . 2807)
  :miscs (15467 . 36265)
  :chars 1833568 :vector 456351 :floats (345 . 328)
  :intervals (604 . 849)
  :strings (138140 . 122806))

After several hours of usage:

(:conses (1558351 . 313757)
  :syms (93011 . 347)
  :miscs (140492 . 28025)
  :chars 9264181 :vector 2283330 :floats (347 . 638)
  :intervals (2771 . 4880)
  :strings (467731 . 34316))

At this point, the total memory occupied was about 400MB, and Emacs 
complained that "memory was exhausted; save buffers and quit" (or 
something to that effect).  I ran garbage-collect, killed all the stray 
buffers and left it around for a while (about 10-15 minutes), and the 
memory came down to 340MB.  (The memory usage figures are from the 
Windows XP Task Manger - "VM Size" - VM here referring to the virtual 
memory.)

After that, I could do a couple of more folder visits and then Emacs 
crashed in the middle of a folder visit.  Come to think of it, almost 
all the Emacs crashes I have experienced over the last year or so were 
probably related to the memory leak problem.  They all happened when I 
was visiting a folder.

Recalling from my memory, these kind of crashes have started happening 
with Emacs 22.3.  There were no such crashes before then.  I stuck to 
Emacs 22.2 for a long time, hoping that whatever was causing the crashes 
would get fixed in due course.  But with every new release, the same 
kind of crashes appeared.

Here is a crash report from yesterday produced by DrMinGW. 
Unfortunately, I was in the middle of a Lisp debugger session and the 
crash happened while the debugger was trying print a large data 
structure.  But this does look like a memory problem to my untrained eye.

Cheers,
Uday


---------------


emacs.exe caused an Access Violation at location 77c47412 in module 
msvcrt.dll Writing to location 18874000.

Registers:
eax=00829240 ebx=02bb2802 ecx=00000000 edx=00000001 esi=008292bf 
edi=18874000
eip=77c47412 esp=00829290 ebp=00829298 iopl=0         nv up ei ng nz ac 
pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000 
efl=00000293

Call stack:
77C47412  msvcrt.dll:77C47412  memmove
0110AD0F  emacs.exe:0110AD0F  printchar  print.c:323

0110FC03  emacs.exe:0110FC03  print_object  print.c:1645

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0111148E  emacs.exe:0111148E  print_object  print.c:2215

0111148E  emacs.exe:0111148E  print_object  print.c:2215

011105A3  emacs.exe:011105A3  print_object  print.c:1921

0110EAE2  emacs.exe:0110EAE2  print  print.c:1305

0110CF92  emacs.exe:0110CF92  Fprin1  print.c:751

01022B04  emacs.exe:01022B04  Ffuncall  eval.c:3034

01114C11  emacs.exe:01114C11  Fbyte_code  bytecode.c:680

010232C3  emacs.exe:010232C3  funcall_lambda  eval.c:3220

01022D03  emacs.exe:01022D03  Ffuncall  eval.c:3077

01114C11  emacs.exe:01114C11  Fbyte_code  bytecode.c:680

01021B46  emacs.exe:01021B46  Feval  eval.c:2356

0101ED68  emacs.exe:0101ED68  Fprogn  eval.c:415

01102A41  emacs.exe:01102A41  Fsave_window_excursion  window.c:6575

0111579A  emacs.exe:0111579A  Fbyte_code  bytecode.c:841

010232C3  emacs.exe:010232C3  funcall_lambda  eval.c:3220

01022D03  emacs.exe:01022D03  Ffuncall  eval.c:3077

010221D6  emacs.exe:010221D6  Fapply  eval.c:2506

010225FD  emacs.exe:010225FD  apply1  eval.c:2780

0101EB72  emacs.exe:0101EB72  call_debugger  eval.c:272

01022E12  emacs.exe:01022E12  Ffuncall  eval.c:3102

01114C11  emacs.exe:01114C11  Fbyte_code  bytecode.c:680

01021B46  emacs.exe:01021B46  Feval  eval.c:2356

01020165  emacs.exe:01020165  internal_catch  eval.c:1228

0111580E  emacs.exe:0111580E  Fbyte_code  bytecode.c:855

010232C3  emacs.exe:010232C3  funcall_lambda  eval.c:3220

01022D03  emacs.exe:01022D03  Ffuncall  eval.c:3077

010221D6  emacs.exe:010221D6  Fapply  eval.c:2506

010225FD  emacs.exe:010225FD  apply1  eval.c:2780

01117B84  emacs.exe:01117B84  Fcall_interactively  callint.c:396

01022B2E  emacs.exe:01022B2E  Ffuncall  eval.c:3037

010226CC  emacs.exe:010226CC  call3  eval.c:2857

01014D52  emacs.exe:01014D52  Fcommand_execute  keyboard.c:10562

010150C3  emacs.exe:010150C3  Fexecute_extended_command  keyboard.c:10675

01022AE3  emacs.exe:01022AE3  Ffuncall  eval.c:3031

0111932A  emacs.exe:0111932A  Fcall_interactively  callint.c:869

01022B2E  emacs.exe:01022B2E  Ffuncall  eval.c:3037

010226CC  emacs.exe:010226CC  call3  eval.c:2857

01014D52  emacs.exe:01014D52  Fcommand_execute  keyboard.c:10562

010076E0  emacs.exe:010076E0  command_loop_1  keyboard.c:1914

01020614  emacs.exe:01020614  internal_condition_case  eval.c:1492

01005FE3  emacs.exe:01005FE3  command_loop_2  keyboard.c:1362

01020165  emacs.exe:01020165  internal_catch  eval.c:1228

01005F9C  emacs.exe:01005F9C  command_loop  keyboard.c:1342

0100588E  emacs.exe:0100588E  recursive_edit_1  keyboard.c:956

010059F7  emacs.exe:010059F7  Frecursive_edit  keyboard.c:1019

010028F5  emacs.exe:010028F5  main  emacs.c:1835

010010DB  emacs.exe:010010DB
01001178  emacs.exe:01001178
010036D0  emacs.exe:010036D0  _start  unexw32.c:131

7C817077  kernel32.dll:7C817077  RegisterWaitForInputIdle

Re: Memory leaks

[Eli Zaretskii]

> From: Uday S Reddy <u.s.reddy@cs.bham.ac.uk>
> Date: Fri, 04 Mar 2011 11:39:05 +0000
> Cc: u.s.reddy@cs.bham.ac.uk
> 
> Initial:
> 
> (:conses (102590 . 13880)
>   :syms (20328 . 54)
>   :miscs (830 . 152)
>   :chars 570345 :vector 329747 :floats (72 . 111)
>   :intervals (348 . 189)
>   :strings (21259 . 5705))
> 
> After one visit to my INBOX folder (22MB) and quitting:
> 
> (:conses (663630 . 605600)
>   :syms (32257 . 2608)
>   :miscs (15443 . 18022)
>   :chars 1639236 :vector 454670 :floats (345 . 261)
>   :intervals (503 . 622)
>   :strings (99122 . 90256))
> 
> After a second visit to the same folder and quitting:
> 
> (:conses (1113362 . 334402)
>   :syms (32264 . 2807)
>   :miscs (15467 . 36265)
>   :chars 1833568 :vector 456351 :floats (345 . 328)
>   :intervals (604 . 849)
>   :strings (138140 . 122806))

Is this with Rmail or with some other mail agent?

With Rmail, I cannot reproduce this with the current trunk.  Here are
my results of garbage-collect, the first one is the initial one, the
others are pairs, where the first one is after visiting my 15MB INBOX,
the second one after killing the RMAIL buffer:

((49547 . 8089) (14061 . 0) (21 . 113) 63254 292573 (44 . 78) (322 . 66) (3574 . 2033))

((69953 . 4823) (16430 . 0) (2030 . 3583) 219252 322528 (49 . 73) (373 . 128) (10009 . 4103))

((67776 . 6125) (16430 . 0) (33 . 246) 217261 316518 (49 . 73) (365 . 136) (10006 . 4106))

((70073 . 6199) (16434 . 0) (2032 . 3089) 219400 322528 (49 . 73) (388 . 164) (10016 . 4096))

((67867 . 6905) (16434 . 0) (35 . 340) 217317 316510 (49 . 73) (379 . 137) (10010 . 4102))

((70162 . 6779) (16438 . 0) (2034 . 3007) 219418 322528 (49 . 73) (400 . 167) (10018 . 4094))

((67956 . 7360) (16438 . 0) (37 . 342) 217335 316510 (49 . 73) (391 . 176) (10012 . 4100))

You can see that the counters go up very slowly, much slower than what
you report.

In fact, the initial report in my cases shows counters that are
several times smaller than in your case.  Can you explain this
discrepancy?

I see similar numbers in Emacs 23.3, so it's not like the trunk is
very different.

Re: Memory leaks

[Uday S Reddy]

Eli Zaretskii writes:

> Is this with Rmail or with some other mail agent?

Hi Eli, my figures for VM, the mail agent that I maintain.  It is much
more of a feature-laden system.  So, it probably uses more memory.
But the problem I am after is the one that the memory is leaking, it
is not getting reused.

I don't know yet whether it is a VM problem or an Emacs problem.  I am
still trying to understand what is going on.

Any tips would be great!

Cheers,
Uday

Re: Memory leaks

[Uday S Reddy]

On 3/4/2011 11:39 AM, Uday S Reddy wrote:

> After several hours of usage:
>
> (:conses (1558351 . 313757)
> :syms (93011 . 347)
> :miscs (140492 . 28025)
> :chars 9264181 :vector 2283330 :floats (347 . 638)
> :intervals (2771 . 4880)
> :strings (467731 . 34316))

One of the things I have decided to check is what I am doing with 
vectors because more than 2M vector slots were being locked up in an 
empty Emacs session.

A widespread use of vectors in VM is for obarrays.  The code might look 
something like this:

    (let ((obarray (make-vector 67 0)))
       ... intern lots of symbols in obarray
       (setq buffer-local-var obarray))

I have begun to wonder if obarray gets freed after buffer-local-var 
stops pointing to it, say through another setq to buffer-local-var.

Am I supposed to unintern the symbols to free up their memory?

Cheers,
Uday

Re: Memory leaks

[Andreas Schwab]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> I have begun to wonder if obarray gets freed after buffer-local-var stops
> pointing to it, say through another setq to buffer-local-var.

Obarrays are ordinary vectors, that get garbage collected like any other
object.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leaks

[David Kastrup]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:
>
>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>> pointing to it, say through another setq to buffer-local-var.
>
> Obarrays are ordinary vectors, that get garbage collected like any other
> object.

Unfortunately, symbols are daisy-chained from their obarray bucket, so
even unused symbols can't be collected until all their successors in the
hidden hash bucket chain have become unused as well.

-- 
David Kastrup

Re: Memory leaks

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:
>>
>>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>>> pointing to it, say through another setq to buffer-local-var.
>>
>> Obarrays are ordinary vectors, that get garbage collected like any other
>> object.
>
> Unfortunately, symbols are daisy-chained from their obarray bucket, so
> even unused symbols can't be collected until all their successors in
> the hidden hash bucket chain have become unused as well.

Their predecessors I mean.

-- 
David Kastrup

Re: Memory leaks

[Uday S Reddy]

On 3/7/2011 8:33 PM, Andreas Schwab wrote:
> Uday S Reddy<u.s.reddy@cs.bham.ac.uk>  writes:
>
>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>> pointing to it, say through another setq to buffer-local-var.
>
> Obarrays are ordinary vectors, that get garbage collected like any other
> object.

I didn't think the vectors themselves would be in trouble, but rather 
what is behind the vectors: the buckets, the symbols and so on.  Even 
after the vectors are gone, there could be references to the symbols in 
other places, could there not?  So, it is not entirely clear when all 
that memory could be collected?

Cheers,
Uday

Re: Memory leaks

[Stefan Monnier]

>>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>>> pointing to it, say through another setq to buffer-local-var.
>> Obarrays are ordinary vectors, that get garbage collected like any other
>> object.
> I didn't think the vectors themselves would be in trouble, but rather what
> is behind the vectors: the buckets, the symbols and so on.  Even after the
> vectors are gone, there could be references to the symbols in other places,
> could there not?  So, it is not entirely clear when all that memory could
> be collected?

But then you'd have lots of symbols, not lots of vectors.


        Stefan

Re: Memory leaks

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> I have begun to wonder if obarray gets freed after buffer-local-var stops
>>>> pointing to it, say through another setq to buffer-local-var.
>>> Obarrays are ordinary vectors, that get garbage collected like any other
>>> object.
>> I didn't think the vectors themselves would be in trouble, but rather what
>> is behind the vectors: the buckets, the symbols and so on.  Even after the
>> vectors are gone, there could be references to the symbols in other places,
>> could there not?  So, it is not entirely clear when all that memory could
>> be collected?
>
> But then you'd have lots of symbols, not lots of vectors.

Unless those symbols have lots of vectors as some of their values.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leaks

[Richard Stallman]

    Unfortunately, symbols are daisy-chained from their obarray bucket, so
    even unused symbols can't be collected until all their successors in the
    hidden hash bucket chain have become unused as well.

It would not be hard to make GC know how to GC symbols and delete them
from the obarray.  This is a traditional Lisp feature that I left out
to make Emacs smaller.

However, it won't do much good unless you create a lot of symbols that
you don't need for long.  Is that happening?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: Memory leaks

[Richard Stallman]

Is it possible to write code to record which vectors were consed by
the operation suspected of having a memory leak?  Then a function to
call from GDB to test whether a vector are still alive.

After you find one, you could do a GC with a conditional breakpoint
to stop when that vector is marked.  Then you would see exactly
what is keeping it alive.  If you do this for each vector that was
consed during the operation in question, you will eventually find
one that was expected to be GC'd, and find the bug.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: Memory leaks

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> Is it possible to write code to record which vectors were consed by
> the operation suspected of having a memory leak?  Then a function to
> call from GDB to test whether a vector are still alive.

It should be possible to record them in a key-weak hashtable.  If they
don't leave after explicitly calling the garbage collector, you've got a
leak.

-- 
David Kastrup

memory leaks

[Madhu]

There is obviously some memory leak in emacs and I don't know how to
trace the culprit.  Emacs lops up 2GB RSS: ps axuw |grep emacs

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
madhu     5434  6.9 54.2 2235612 2115564 ?     SNsl Sep19  66:42 emacs --daemon --debug-init

(garbage-collect) shows

((conses 16 981544 136697)
 (symbols 48 56864 5)
 (strings 32 312273 12669)
 (string-bytes 1 18002558)
 (vectors 16 68285)
 (vector-slots 8 1780950 150102)
 (floats 8 1089 2405)
 (intervals 56 6605 2469)
 (buffers 992 79))

So does this look like a X fonts issue?  How can I figure out which
package is the culprit.

1. Emacs has a bug where I can't look at /proc/5434/emacs from within
emacs it will only read 16384 bytes from the proc filesystem

2. Trying to Unload a feature hits all sorts of design and
implementation bugs with the cl-generics scourge which is unfortunately
used to implement emacs.  Is unloading features likely to help at all in
reducing the memory and getting a clue about the errant package?

Re: memory leaks

[Jean-Christophe Helary]

I had something similar happening the other day and Eli told me it was probably fixed with some code recently committed to master. Have you tried a recent build ?

Jean-Christophe Helary 

> On Sep 20, 2019, at 16:23, Madhu <enometh@meer.net> wrote:
> 
> 
> There is obviously some memory leak in emacs and I don't know how to
> trace the culprit.  Emacs lops up 2GB RSS: ps axuw |grep emacs
> 
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> madhu     5434  6.9 54.2 2235612 2115564 ?     SNsl Sep19  66:42 emacs --daemon --debug-init

Jean-Christophe Helary
-----------------------------------------------
http://mac4translators.blogspot.com @brandelune

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Fri, 20 Sep 2019 12:53:46 +0530
> 
> 
> There is obviously some memory leak in emacs and I don't know how to
> trace the culprit.  Emacs lops up 2GB RSS: ps axuw |grep emacs
> 
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> madhu     5434  6.9 54.2 2235612 2115564 ?     SNsl Sep19  66:42 emacs --daemon --debug-init

Which version of Emacs?  If this is a development snapshot, when was
it built?

> 1. Emacs has a bug where I can't look at /proc/5434/emacs from within
> emacs it will only read 16384 bytes from the proc filesystem

Please report as a bug using "M-x report-emacs-bug".

> 2. Trying to Unload a feature hits all sorts of design and
> implementation bugs with the cl-generics scourge which is unfortunately
> used to implement emacs.  Is unloading features likely to help at all in
> reducing the memory and getting a clue about the errant package?

Likewise.

Re: memory leaks

[Madhu]

* Eli Zaretskii <mailman.572.1568967069.2190.help-gnu-emacs@gnu.org> :
Wrote on Fri, 20 Sep 2019 11:11:11 +0300:

>> From: Madhu <enometh@meer.net>
>> Date: Fri, 20 Sep 2019 12:53:46 +0530
>> There is obviously some memory leak in emacs and I don't know how to
>> trace the culprit.  Emacs lops up 2GB RSS: ps axuw |grep emacs
>>
>> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>> madhu 5434 6.9 54.2 2235612 2115564 ?  SNsl Sep19 66:42 emacs
>> --daemon --debug-init
>
> Which version of Emacs?  If this is a development snapshot, when was
> it built?

Sorry I neglected to include that info: it is master from around
20-Aug-2019: (commit 08d7cab)

GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw scroll
bars) of 2019-08-20

If there isn't a known problem I'd still appreciate a clue or two in
trying to figure out what the problem is.

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Fri, 20 Sep 2019 14:01:05 +0530
> 
> >> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> >> madhu 5434 6.9 54.2 2235612 2115564 ?  SNsl Sep19 66:42 emacs
> >> --daemon --debug-init
> >
> > Which version of Emacs?  If this is a development snapshot, when was
> > it built?
> 
> Sorry I neglected to include that info: it is master from around
> 20-Aug-2019: (commit 08d7cab)

Then what you see is almost certainly a problem with GC that has been
solved recently.  Update from Git and rebuild.

Re: memory leaks

[Madhu]

* Eli Zaretskii <mailman.578.1568971399.2190.help-gnu-emacs@gnu.org> :
Wrote on Fri, 20 Sep 2019 12:21:23 +0300:

>> From: Madhu <enometh@meer.net>
>> Date: Fri, 20 Sep 2019 14:01:05 +0530
>>
>> >> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>> >> madhu 5434 6.9 54.2 2235612 2115564 ?  SNsl Sep19 66:42 emacs
>> >> --daemon --debug-init
>> >
>> > Which version of Emacs?  If this is a development snapshot, when was
>> > it built?
>>
>> Sorry I neglected to include that info: it is master from around
>> 20-Aug-2019: (commit 08d7cab)
>
> Then what you see is almost certainly a problem with GC that has been
> solved recently.  Update from Git and rebuild.

One year on I'm having the same problems on master (and I've been
noticing it for a few months now)

virtual memory reports 2.9G resident 2.5G shared 10.9M.

$ ps o pid,rss,drs,sz,share,start_time,vsize,cmd 26285

  PID   RSS   DRS  SIZE - START    VSZ CMD
26285 2643236 2996379 2664940 - Sep09 2998948 /7/gtk/emacs/build-xt-xft/src/emacs --debug-init --daemon

GC reports
Memory information:
((conses 16 3591911 2541685)
 (symbols 48 87049 452)
 (strings 32 528119 452566)
 (string-bytes 1 30189681)
 (vectors 16 217149)
 (vector-slots 8 3232842 6057920)
 (floats 8 1637 5252)
 (intervals 56 501483 50429)
 (buffers 992 581))

which is a small fraction of what emacs is using.

This is on a recent emacs from master
In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d
scroll bars)  of 2020-09-06 built on maher
Base Repository revision: 6fc502c1ef327ab357c971b9bffbbd7cb6a436f1
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000


Perhaps there is already a bug report for this on debbugs?

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Sat, 12 Sep 2020 18:32:11 +0530
> 
> One year on I'm having the same problems on master (and I've been
> noticing it for a few months now)
> 
> virtual memory reports 2.9G resident 2.5G shared 10.9M.

Fascinating.  Why no one else has these problems, I wonder.

> Perhaps there is already a bug report for this on debbugs?

I don't think so, so please file a bug report with all the details
(mainly how do you get from starting Emacs to such a huge RSS).

Re: memory leaks

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > virtual memory reports 2.9G resident 2.5G shared 10.9M.
>
> Fascinating.  Why no one else has these problems, I wonder.

I have the problem.  From time to time I see that my memory is getting
out and Emacs uses 4 GB of it.  I use to restart Emacs then.

Michael.

Re: memory leaks

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sat, 12 Sep 2020 15:55:33 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > virtual memory reports 2.9G resident 2.5G shared 10.9M.
> >
> > Fascinating.  Why no one else has these problems, I wonder.
> 
> I have the problem.  From time to time I see that my memory is getting
> out and Emacs uses 4 GB of it.  I use to restart Emacs then.

Then I wonder how come you didn't report it earlier, with all the
details.  Please do that now, TIA.

Re: memory leaks

[Igor Sosa Mayor]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Madhu <enometh@meer.net>
>> Date: Sat, 12 Sep 2020 18:32:11 +0530
>> 
>> One year on I'm having the same problems on master (and I've been
>> noticing it for a few months now)
>> 
>> virtual memory reports 2.9G resident 2.5G shared 10.9M.
>
> Fascinating.  Why no one else has these problems, I wonder.

[...]

I am also experiencing a similar problem (version 27.1 in archlinux)...

Re: memory leaks

[Eli Zaretskii]

> From: Igor Sosa Mayor <joseleopoldo1792@gmail.com>
> Date: Sat, 12 Sep 2020 21:14:01 +0200
> 
> I am also experiencing a similar problem (version 27.1 in archlinux)...

Then I'd appreciate if everyone who is experiencing these problems
will please describe them in a full-fledged bug report, including how
long it takes to get to such large RSS values from the start.

Please don't leave such problems unreported.  They will not fix
themselves.

TIA

Re: memory leaks

[Igor Sosa Mayor]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Igor Sosa Mayor <joseleopoldo1792@gmail.com>
>> Date: Sat, 12 Sep 2020 21:14:01 +0200
>> 
>> I am also experiencing a similar problem (version 27.1 in archlinux)...
>
> Then I'd appreciate if everyone who is experiencing these problems
> will please describe them in a full-fledged bug report, including how
> long it takes to get to such large RSS values from the start.
>
> Please don't leave such problems unreported.  They will not fix
> themselves.

Of course. But I'm doing some preliminary research on my own, because I
thought, maybe it is a problem with one of the hundreds of packages I
load...

Re: memory leaks

[Eli Zaretskii]

> From: Igor Sosa Mayor <joseleopoldo1792@gmail.com>
> Date: Sat, 12 Sep 2020 21:28:27 +0200
> 
> > Please don't leave such problems unreported.  They will not fix
> > themselves.
> 
> Of course. But I'm doing some preliminary research on my own, because I
> thought, maybe it is a problem with one of the hundreds of packages I
> load...

Thank you.

Re: memory leaks

[Madhu]

* Eli Zaretskii <83imcidcfp.fsf@gnu.org> :
Wrote on Sat, 12 Sep 2020 22:26:02 +0300:
> Then I'd appreciate if everyone who is experiencing these problems
> will please describe them in a full-fledged bug report, including how
> long it takes to get to such large RSS values from the start.
>
> Please don't leave such problems unreported.  They will not fix
> themselves.

The problem is that there is nothing to report except that the RSS is
growing unreasonably.  In normal operation with all my packages loaded
emacs wouldn't go beyond around 500M RSS even when running for a week.

I usually only notice the leak when it has gone beyond 2G - when linux
refuses to suspend because I have limited swap.  In most cases emacs
would be running for a few days.  There are over 600 `features' in these
emacs but the major memory hog packages I was using are magit, sly,
gnus, and ivy - but I don't think these can be the culprits.

ATM I have a gtk emacs instance that is running at ~900M RSS, which may
be an early stage.  If you have any suggestions on how to go about
debugging GC - it would help in submitting a useful bug repoprt. But as
of now I have nothing more than "it leaks"

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Sun, 13 Sep 2020 08:47:29 +0530
> 
> > Please don't leave such problems unreported.  They will not fix
> > themselves.
> 
> The problem is that there is nothing to report except that the RSS is
> growing unreasonably.  In normal operation with all my packages loaded
> emacs wouldn't go beyond around 500M RSS even when running for a week.

When you file a bug report, Emacs collects important information about
your build and the environment in which you run Emacs.  That already
tells the Emacs developers a lot, and can give ideas regarding the
possible reasons.

Then I suggest to tell in your report how many buffers you have in
that session, and what is their total size.

And finally, I'd encourage you to try to describe your use patterns,
especially regarding operations that could require significant amounts
of memory: visiting large file, loading many fonts, creating large
buffers that don't visit files, etc.  If you can afford keeping notes
about what you did immediately prior to RSS becoming significantly
larger, that is even better.

> ATM I have a gtk emacs instance that is running at ~900M RSS, which may
> be an early stage.  If you have any suggestions on how to go about
> debugging GC - it would help in submitting a useful bug repoprt.

I don't think the problem is in GC, and the information about that is
part of the info collected by report-emacs-bug anyway.

Thanks.

Re: memory leaks

[Stefan Monnier]

> The problem is that there is nothing to report except that the RSS is
> growing unreasonably.  In normal operation with all my packages loaded
> emacs wouldn't go beyond around 500M RSS even when running for a week.

The `garbage-collect` command actually can return info about the current
Lisp heap use.  The `memory-usage` package can display it in a more
user-friendly format.

That can be very useful to see if those 500MB belong to the Lisp heap or
to ... something else (typically libraries, like those of the GUI or
those used to display images).


        Stefan

Re: memory leaks

[Madhu]

* Stefan Monnier <jwvh7s1780o.fsf-monnier+emacs@gnu.org> :
Wrote on Sun, 13 Sep 2020 16:14:36 -0400:
> The `garbage-collect` command actually can return info about the current
> Lisp heap use.  The `memory-usage` package can display it in a more
> user-friendly format.
>
> That can be very useful to see if those 500MB belong to the Lisp heap or
> to ... something else (typically libraries, like those of the GUI or
> those used to display images).

The numbers I reported by garbage-collect do not account for the memory
usage. FWIW I've posted that info again at
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395

In 2019 i suspected the culprit was the fonts system.  It doesn't seem
to be toolkit related. I don't use emacs as an imageviewer (but I have
to check that). Since the interest in CONFIG_PSI there are a number of
tools on github that monitor memory - it should be possible to come up
with some tool that alerts emacs of memory surges while keeping some
history.

Re: memory leaks

[Madhu]

* Madhu <m35z8gise6.fsf@leonis4.robolove.meer.net> :
Wrote on Mon, 14 Sep 2020 15:36:57 +0530:
> The numbers I reported by garbage-collect do not account for the memory
> usage. FWIW I've posted that info again at
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395
>
> In 2019 i suspected the culprit was the fonts system.  It doesn't seem
> to be toolkit related. I don't use emacs as an imageviewer (but I have
> to check that). Since the interest in CONFIG_PSI there are a number of
> tools on github that monitor memory - it should be possible to come up
> with some tool that alerts emacs of memory surges while keeping some
> history.

I found test case for this bug and I sent mail to 43395@debbugs.gnu.org,
and qmail told me that the mail was accepted:
delivery 1: success:
209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
But the bugreport didn't show up on gnu.emacs.bug.

Here is the test case:

#+BEGIN_SRC
$ cat > f.c
#include <stdio.h>
int
main()
{
  char c = ' ';
  while (c != 'q' && c != 'Q')
    {
      fprintf(stdout, "Press q then enter to quit: ");
      c = fgetc(stdin);
    }
  return 0;
}
^D

$ gcc f.c
$ emacs -Q
#+END_SRC

M-x shell-command ./a.out

Then the process hangs. But Emacs' memory grows unbounded.

WARNING. Be careful to interrupt it with ^G before the OOM killer
kicks in.  If you have swap you may lose control of your machine
indefinitely.

After you interrupt the sub process, Emacs is left with unreclaimed
gigabytes of RSS

Please let me know if you can reproduce this

Re: memory leaks

[Madhu]

[Test case follows. Sorry Drew for trying to post to help-gnu and
debbugs simultaneously, but this is my third attempt at sending this
test case - I've tried sending it to debbugs.gnu.org and to
gmane.emacs.help newsgroup - and my messages have been dropped]


* Madhu <m35z8gise6.fsf@leonis4.robolove.meer.net> :
Wrote on Mon, 14 Sep 2020 15:36:57 +0530:
> The numbers I reported by garbage-collect do not account for the memory
> usage. FWIW I've posted that info again at
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395

I found test case for this bug and I sent mail to 43395@debbugs.gnu.org,
and qmail told me that the mail was accepted:
delivery 1: success:
209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
But the bugreport didn't show up on gnu.emacs.bug.

Here is the test case:

#+BEGIN_SRC
$ cat > f.c
#include <stdio.h>
int
main()
{
  char c = ' ';
  while (c != 'q' && c != 'Q')
    {
      fprintf(stdout, "Press q then enter to quit: ");
      c = fgetc(stdin);
    }
  return 0;
}
^D

$ gcc f.c
$ emacs -Q
#+END_SRC

M-x shell-command ./a.out

Then the process hangs. But Emacs' memory grows unbounded.

WARNING. Be careful to interrupt it with ^G before the OOM killer
kicks in.  If you have swap you may lose control of your machine
indefinitely.

After you interrupt the sub process, Emacs is left with unreclaimed
gigabytes of RSS

Please let me know if you can reproduce this

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Tue, 10 Nov 2020 06:54:56 +0530
> Cc: 43395@debbugs.gnu.org
> 
> #+BEGIN_SRC
> $ cat > f.c
> #include <stdio.h>
> int
> main()
> {
>   char c = ' ';
>   while (c != 'q' && c != 'Q')
>     {
>       fprintf(stdout, "Press q then enter to quit: ");
>       c = fgetc(stdin);
>     }
>   return 0;
> }
> ^D
> 
> $ gcc f.c
> $ emacs -Q
> #+END_SRC
> 
> M-x shell-command ./a.out
> 
> Then the process hangs. But Emacs' memory grows unbounded.

Short answer: you use a command that cannot handle this case.  For the
details, please refer to the bug tracker.

Re: memory leaks

[Madhu]

| Your bug report #43389: 28.0.50; memory leak which was filed against the emacs package, has been closed.

Kudos!

Thanks Trevor and everyone else for tracking this down.  Your hard work
is much appreciated.

The debbugs report #43399 rendered under eww runs to over 100,800 words,
much longer than your average murder mystery; where the clue to the
culprit isn't given till near end.  It is a great relief to see that
justice is finally done and ones faith in humanity is restored.

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Mon, 09 Nov 2020 23:20:55 +0530
> 
> I found test case for this bug and I sent mail to 43395@debbugs.gnu.org,
> and qmail told me that the mail was accepted:
> delivery 1: success:
> 209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
> But the bugreport didn't show up on gnu.emacs.bug.

That bug is archived, so any messages to it are discarded.  You need
to unarchive it first (see admin/notes/bugtracker for the details).

> Here is the test case:
> 
> #+BEGIN_SRC
> $ cat > f.c
> #include <stdio.h>
> int
> main()
> {
>   char c = ' ';
>   while (c != 'q' && c != 'Q')
>     {
>       fprintf(stdout, "Press q then enter to quit: ");
>       c = fgetc(stdin);
>     }
>   return 0;
> }
> ^D
> 
> $ gcc f.c
> $ emacs -Q
> #+END_SRC
> 
> M-x shell-command ./a.out
> 
> Then the process hangs. But Emacs' memory grows unbounded.
> 
> WARNING. Be careful to interrupt it with ^G before the OOM killer
> kicks in.  If you have swap you may lose control of your machine
> indefinitely.
> 
> After you interrupt the sub process, Emacs is left with unreclaimed
> gigabytes of RSS
> 
> Please let me know if you can reproduce this

This was already discussed in that bug, and you said back then that
when the shell buffer is killed, the memory is freed up.  So how come
you now say this is still a bug, and what is the difference between
the case discussed in that bug and what you describe now?

Re: memory leaks

[Madhu]

* Eli Zaretskii <83ee3ihs17.fsf@gnu.org> :
Wrote on Fri, 04 Mar 2022 13:32:36 +0200:
> This was already discussed in that bug, and you said back then that
> when the shell buffer is killed, the memory is freed up.  So how come
> you now say this is still a bug, and what is the difference between
> the case discussed in that bug and what you describe now?

That message was from nov 2020. It is a duplicate

I onky authorized gmane to send my message this week to emacs-help,
looks like I had a bunch of messages posted through gmane over the years
which were not posted by gmane because i was not authorized, which have
now been delivered, but which were kept in hold. I think in most cases i
posted the messages outside of gmane.

sorry for the noise

I've not seen the memory "leaks" recently. Maybe I stopped using the
packages that triggered them

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Sun, 06 Mar 2022 16:29:41 +0530
> 
> * Eli Zaretskii <83ee3ihs17.fsf@gnu.org> :
> Wrote on Fri, 04 Mar 2022 13:32:36 +0200:
> > This was already discussed in that bug, and you said back then that
> > when the shell buffer is killed, the memory is freed up.  So how come
> > you now say this is still a bug, and what is the difference between
> > the case discussed in that bug and what you describe now?
> 
> That message was from nov 2020. It is a duplicate
> 
> I onky authorized gmane to send my message this week to emacs-help,
> looks like I had a bunch of messages posted through gmane over the years
> which were not posted by gmane because i was not authorized, which have
> now been delivered, but which were kept in hold. I think in most cases i
> posted the messages outside of gmane.
> 
> sorry for the noise
> 
> I've not seen the memory "leaks" recently. Maybe I stopped using the
> packages that triggered them

OK, thanks.  And sorry for not noticing the old date.