bug#9221: Memory leak

bug#9221: Memory leak

[Stefan Monnier]

I'm seeing some weird behavior linked to insane memory consumption.
E.g. my Gnus session tends to grow to more than 2GB and then become
unusage and unkillable (or close enough: in D state, "kill -9" isn't
enough to make it stop use CPU, tho I guess that CPU use is really due
to something like the OS being in the process of dumping the core file,
tho I don't see any left over core files).

I did manage to attach to it and get a few backtraces (with
a breakpoint on mmap) before it was too late, and bidi_shelve_cache
showed up in most of the backtraces (like 5 out of 8, maybe): not
a strong indictment, but at least a lead worth following until I get
more data.


        Stefan




In GNU Emacs 24.0.50.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2011-08-01 on ceviche
Windowing system distributor `The X.Org Foundation', version 11.0.11002000
configured using `configure  'CFLAGS=-Wall -Wno-pointer-sign -DUSE_LISP_UNION_TYPE -DSYNC_INPUT -DENABLE_CHECKING -DXASSERTS -DFONTSET_DEBUG -g -O1 -I/usr/include/GNUstep' '--enable-maintainer-mode' '--with-x-toolkit=lucid''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: fr_CH.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: InactiveMinibuffer

Minor modes in effect:
  shell-dirtrack-mode: t
  electric-pair-mode: t
  electric-indent-mode: t
  url-handler-mode: t
  global-reveal-mode: t
  reveal-mode: t
  auto-insert-mode: t
  savehist-mode: t
  minibuffer-electric-default-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
p C-e <left> <M-backspace> S e p C-a C-x C-s C-c C-c 
<return> <down> <left> <return> \ n e w c o M-/ SPC 
\ c h a p t e r <backspace> <backspace> <backspace> 
i t r e SPC { c h a p t e r C-e <return> \ n e w c 
o m m a n d SPC \ M a y SPC { M a y C-e C-a C-k C-y 
C-y <left> <left> <M-backspace> A u g <left> <left> 
<left> <left> <left> <M-backspace> A u g C-a C-p C-p 
C-k C-n C-n C-y C-x C-s C-c C-c <return> C-c C-c <return> 
<switch-frame> <switch-frame> <help-echo> <help-echo> 
<switch-frame> <switch-frame> <help-echo> <switch-frame> 
<switch-frame> <help-echo> <switch-frame> <switch-frame> 
<help-echo> <right> <right> <right> <right> <down> 
<down> <down> <down> <down> <down> <down> <down> <next> 
<next> <prior> <prior> <next> <next> <help-echo> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
C-s \ J u n <right> <left> { } C-/ M-< <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <left> <left> ~ 
<down> <left> <down> <left> ~ <down> <down> <left> 
<left> ~ <down> <down> <left> <left> ~ <down> <down> 
<left> <left> ~ <down> <left> <right> M-f <right> C-s 
C-w C-s C-a C-x C-s C-c C-c <return> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<help-echo> <prior> <next> <prior> <prior> <prior> 
<next> <next> <next> <help-echo> <switch-frame> <switch-frame> 
<help-echo> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <help-echo> <next> <next> <prior> <help-echo> 
<switch-frame> <switch-frame> <switch-frame> M-x l 
o - l i <tab> <return> a <tab> <return> s m i e <return> 
M-x <tab> C-g C-h f s m i e - r u <tab> <tab> C-g M-x 
r e p o <tab> r <tab> <return>

Recent messages:
Type C-c C-c to toggle between editing or viewing the document.
Warning: interactive-p is obsolete! [8 times]
Making completion list...
Loading smie...done
Making completion list...
Quit
Making completion list...
Quit
Warning: interactive-p is obsolete! [2 times]
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort mail-extr message rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
mail-utils gmm-utils mailheader emacsbug smie cus-edit cus-start
cus-load wid-edit autorevert doc-view jka-compr image-mode dired dabbrev
multi-isearch format-spec reftex-vcr reftex-dcr reftex reftex-vars
tex-mode compile shell pcomplete comint ring latexenc executable
copyright vc-bzr filecache longlines server noutline outline easy-mmode
flyspell ispell eldoc checkdoc regexp-opt thingatpt help-mode view
prog-mode load-dir electric url-handlers url-parse auth-source warnings
eieio byte-opt bytecomp byte-compile cconv macroexp assoc gnus-util
password-cache url-vars mm-util mail-prsvr reveal autoinsert uniquify
advice help-fns advice-preload time-date savehist minibuf-eldef
disp-table cl cl-loaddefs all-autoloads company-autoloads
debbugs-autoloads epoch-view-autoloads js2-mode-autoloads
load-dir-autoloads markchars-autoloads minimap-autoloads muse-autoloads
info easymenu rainbow-mode-autoloads register-list-autoloads
sisu-mode-autoloads uni-confusables-autoloads windresize-autoloads
package tabulated-list proof-site proof-autoloads pg-vars bbdb-autoloads
agda2 tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd
tool-bar dnd fontset image fringe lisp-mode register page newcomment
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer loaddefs button faces
cus-face files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
make-network-process dbusbind dynamic-setting system-font-setting
font-render-setting x-toolkit x multi-tty emacs)

bug#9221: Memory leak

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 01 Aug 2011 22:23:18 -0400
> 
> I'm seeing some weird behavior linked to insane memory consumption.
> E.g. my Gnus session tends to grow to more than 2GB and then become
> unusage and unkillable (or close enough: in D state, "kill -9" isn't
> enough to make it stop use CPU, tho I guess that CPU use is really due
> to something like the OS being in the process of dumping the core file,
> tho I don't see any left over core files).

Does anyone else see similar memory footprint growth?

> I did manage to attach to it and get a few backtraces (with
> a breakpoint on mmap) before it was too late, and bidi_shelve_cache
> showed up in most of the backtraces (like 5 out of 8, maybe): not
> a strong indictment, but at least a lead worth following until I get
> more data.

bidi_shelve_cache is used extensively during redisplay, and it does
allocate memory (which should be almost immediately freed by its
companion bidi_unshelve_cache, or by an explicit xfree call in a few
places).  So the fact that you saw it on the callstack of a breakpoint
in mmap is what I'd expect, not really an evidence of a leak.  To see
if the leak is really due to bidi, run a session with
bidi-display-reordering turned off, and see if there's any change.
Although, AFAIR, some calls to bidi_shelve_cache are not conditioned
on bidi.

FWIW, I'm running a session continuously for several days since the
last rebuild, and don't see any unusual footprint, let alone one that
grows without limits.  But that's a with a code base that is slightly
different from the trunk, and I don't use Gnus, so perhaps some change
done lately (including in bidi.c) is responsible.  What trunk revision
are you running.

If we cannot exclude bidi_shelve_cache from the list of suspects, I
could add debugging code that monitored its allocations and
deallocations.  That would allow us to see if there's non-zero
balance.

bug#9221: Memory leak

[Antoine Levitt]

02/08/11 06:39, Eli Zaretskii
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Mon, 01 Aug 2011 22:23:18 -0400
>> 
>> I'm seeing some weird behavior linked to insane memory consumption.
>> E.g. my Gnus session tends to grow to more than 2GB and then become
>> unusage and unkillable (or close enough: in D state, "kill -9" isn't
>> enough to make it stop use CPU, tho I guess that CPU use is really due
>> to something like the OS being in the process of dumping the core file,
>> tho I don't see any left over core files).
>
> Does anyone else see similar memory footprint growth?

Yes (though it doesn't use all my RAM and I can kill it no problem - I
have 4gb, if that's relevant). I also use gnus. It happened twice, I
don't remember what I was doing the first time, and the second was when
I was using ffap inside ERC to display a link in an external browser.

It all started around the time bidi-display-reordering was turned to t,
although I can't be 100% sure it's related.

bug#9221: Memory leak

[Eli Zaretskii]

> From: Antoine Levitt <antoine.levitt@gmail.com>
> Date: Tue, 02 Aug 2011 13:17:07 +0200
> 
> > Does anyone else see similar memory footprint growth?
> 
> Yes (though it doesn't use all my RAM and I can kill it no problem - I
> have 4gb, if that's relevant). I also use gnus. It happened twice, I
> don't remember what I was doing the first time, and the second was when
> I was using ffap inside ERC to display a link in an external browser.

Thanks.  Does the "happened twice" part mean that not every session
shows memory footprint growth like this?  If not, what exactly
happened twice?

bug#9221: Memory leak

[Antoine Levitt]

02/08/11 13:37, Eli Zaretskii
>> From: Antoine Levitt <antoine.levitt@gmail.com>
>> Date: Tue, 02 Aug 2011 13:17:07 +0200
>> 
>> > Does anyone else see similar memory footprint growth?
>> 
>> Yes (though it doesn't use all my RAM and I can kill it no problem - I
>> have 4gb, if that's relevant). I also use gnus. It happened twice, I
>> don't remember what I was doing the first time, and the second was when
>> I was using ffap inside ERC to display a link in an external browser.
>
> Thanks.  Does the "happened twice" part mean that not every session
> shows memory footprint growth like this?  If not, what exactly
> happened twice?

Sorry, I should have been more clear. Emacs ate a lot of ram and slowed
my computer down to a crawl. But I could still kill emacs properly (C-x
C-c), suggesting that it's not a while(1){malloc();} loop, because I
wouldn't have been able to do that if it was. So I killed it. I ran
another session, and the same problem happened again. Both times, it
happened at seemingly random times.

That being said, my emacs session currently takes up 35% of my RAM (that
is more than 1gb so anormal, and it's an uptime of 14 hours, including
many in suspended state). I'll try and leave the session open for a few
hours, so if you have ideas of commands to run to debug it, it's all
yours.

bug#9221: Memory leak

[Andy Moreton]

On Tue 02 Aug 2011, Eli Zaretskii wrote:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Mon, 01 Aug 2011 22:23:18 -0400
>> 
>> I'm seeing some weird behavior linked to insane memory consumption.
>> E.g. my Gnus session tends to grow to more than 2GB and then become
>> unusage and unkillable (or close enough: in D state, "kill -9" isn't
>> enough to make it stop use CPU, tho I guess that CPU use is really due
>> to something like the OS being in the process of dumping the core file,
>> tho I don't see any left over core files).
>
> Does anyone else see similar memory footprint growth?

Yes. My w32 buildon WinXP also shows ever-growing memory consumption.
The growth rate is slower if I add:

(setq-default bidi-display-reordering nil)

but emacs grows to >1GB anyway - it just takes longer. I haven't
biesected this yet, but I can do so if it helps.

    AndyM

bug#9221: Memory leak

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 03 Aug 2011 00:53:55 +0100
> 
> Yes. My w32 buildon WinXP also shows ever-growing memory consumption.
> The growth rate is slower if I add:
> 
> (setq-default bidi-display-reordering nil)

Do you use Gnus?

> I haven't biesected this yet, but I can do so if it helps.

It will help, yes.  My prime suspect is revision 105208.

TIA

bug#9221: Memory leak

[Andy Moreton]

On Wed 03 Aug 2011, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Wed, 03 Aug 2011 00:53:55 +0100
>> 
>> Yes. My w32 buildon WinXP also shows ever-growing memory consumption.
>> The growth rate is slower if I add:
>> 
>> (setq-default bidi-display-reordering nil)
>
> Do you use Gnus?
>
>> I haven't biesected this yet, but I can do so if it helps.
>
> It will help, yes.  My prime suspect is revision 105208.

Thanks for the hint - I'll try rev 105207 first :-)

    AndyM

bug#9221: Memory leak

[Andy Moreton]

On Wed 03 Aug 2011, Andy Moreton wrote:

> On Wed 03 Aug 2011, Eli Zaretskii wrote:
>
>>> From: Andy Moreton <andrewjmoreton@gmail.com>
>>> Date: Wed, 03 Aug 2011 00:53:55 +0100
>>> 
>>> Yes. My w32 buildon WinXP also shows ever-growing memory consumption.
>>> The growth rate is slower if I add:
>>> 
>>> (setq-default bidi-display-reordering nil)
>>
>> Do you use Gnus?
>>
>>> I haven't biesected this yet, but I can do so if it helps.
>>
>> It will help, yes.  My prime suspect is revision 105208.
>
> Thanks for the hint - I'll try rev 105207 first :-)
>
>     AndyM

I've tried building revs 105207 and 105208, each bootstrapped after
'make clean'.

While I can't be entirely sure, revision 105207 seems to behave
itself even after using gnus for a while.

Revision 105208 seems to show growing memory usage, but nothing like as
fast as I observed before a full rebuild. After running for a few hours
memory usage seems to have stabilised at approx 93MB, and I'm not
seeing the > 1GB runaway consumption from before.

I'll keep trying with newer builds to see if I can find anything that
produces the excessive memory usage again.

    AndyM 

bug#9221: Memory leak

[Andy Moreton]

> On Wed 03 Aug 2011, Andy Moreton wrote:
> I've tried building revs 105207 and 105208, each bootstrapped after
> 'make clean'.
>
> While I can't be entirely sure, revision 105207 seems to behave
> itself even after using gnus for a while.
>
> Revision 105208 seems to show growing memory usage, but nothing like as
> fast as I observed before a full rebuild. After running for a few hours
> memory usage seems to have stabilised at approx 93MB, and I'm not
> seeing the > 1GB runaway consumption from before.
>
> I'll keep trying with newer builds to see if I can find anything that
> produces the excessive memory usage again.

I've been trying to bisect this, but it seems to take a while using
emacs before the excessive memory consumption appears, so the test cycle
is a little slow.

The problems seems to appear somewhere between r105368 (good) and
r105375 (bad).

I'll keep looking at this to make sure I can get consistent results.

    AndyM

bug#9221: Memory leak

[Antoine Levitt]

05/08/11 00:14, Andy Moreton
>> On Wed 03 Aug 2011, Andy Moreton wrote:
>> I've tried building revs 105207 and 105208, each bootstrapped after
>> 'make clean'.
>>
>> While I can't be entirely sure, revision 105207 seems to behave
>> itself even after using gnus for a while.
>>
>> Revision 105208 seems to show growing memory usage, but nothing like as
>> fast as I observed before a full rebuild. After running for a few hours
>> memory usage seems to have stabilised at approx 93MB, and I'm not
>> seeing the > 1GB runaway consumption from before.
>>
>> I'll keep trying with newer builds to see if I can find anything that
>> produces the excessive memory usage again.
>
> I've been trying to bisect this, but it seems to take a while using
> emacs before the excessive memory consumption appears, so the test cycle
> is a little slow.

You can use Eli's patch (on emacs-devel) to print
bidi_cache_total_alloc, that's faster.

>
> The problems seems to appear somewhere between r105368 (good) and
> r105375 (bad).
>
> I'll keep looking at this to make sure I can get consistent results.
>
>     AndyM

Is word-wrap t? If yes, does the problem go away when turning it off?

bug#9221: Memory leak

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Thu, 04 Aug 2011 23:14:11 +0100
> 
> I've been trying to bisect this, but it seems to take a while using
> emacs before the excessive memory consumption appears, so the test cycle
> is a little slow.

Bug squashed, I think.  Please try revision 105407.  If it solves the
problem, there's no need to continue bisecting.

> The problems seems to appear somewhere between r105368 (good) and
> r105375 (bad).

If my fix is correct, the offending commit was actually 105208.

> I'll keep looking at this to make sure I can get consistent results.

If the problem does not go away with the current trunk, please do, and
thanks.  Please also reopen this bug if you find excessive memory
consumption with today's trunk.

Thanks a lot for your help!  And thanks to Antoine for finding the
easy way of reproducing it.