Memory leak

Memory leak

[Stefan Monnier]

We have a pretty serious memory leak.
My Emacs session recently died with a memory full situation (the
process was around 500MB VSZ).  So I took a look around and it seem that
every time I create a new frame, my Emacs session grows by around 10MB
and deleting the frame does not recover this memory.
This memory is not accounted for by the counts returned by
`garbage-collect'.

I'm not sure when this problem got introduced, tho the new font code is
a likely culprit.


        Stefan

Re: Memory leak

[David Kastrup]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> We have a pretty serious memory leak.
> My Emacs session recently died with a memory full situation (the
> process was around 500MB VSZ).  So I took a look around and it seem that
> every time I create a new frame, my Emacs session grows by around 10MB
> and deleting the frame does not recover this memory.
> This memory is not accounted for by the counts returned by
> `garbage-collect'.
>
> I'm not sure when this problem got introduced, tho the new font code is
> a likely culprit.

Just to rule out something obvious (pure space overflow):

M-: (garbage-collect) RET

does not return nil, does it?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Memory leak

[Lennart Borgman (gmail)]

David Kastrup wrote:
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> 
>> We have a pretty serious memory leak.
>> My Emacs session recently died with a memory full situation (the
>> process was around 500MB VSZ).  So I took a look around and it seem that
>> every time I create a new frame, my Emacs session grows by around 10MB
>> and deleting the frame does not recover this memory.
>> This memory is not accounted for by the counts returned by
>> `garbage-collect'.
>>
>> I'm not sure when this problem got introduced, tho the new font code is
>> a likely culprit.
> 
> Just to rule out something obvious (pure space overflow):
> 
> M-: (garbage-collect) RET
> 
> does not return nil, does it?

I can see a memory leak on w32 of around 240 K in the above operation 
with a build from 2008-03-01, but not with a build from 2008-01-21 (no 
build in between starts ok).

(garbage-collect) does not return nil with any of the builds above.

PS: Is there any command line utility that shows memory usage on w32?

Re: Memory leak

[Eli Zaretskii]

> Date: Fri, 16 May 2008 02:25:50 +0200
> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>, emacs-devel@gnu.org
> 
> PS: Is there any command line utility that shows memory usage on w32?

Download PsTools, and then say "pslist -m emacs".

Re: Memory leak

[Juanma Barranquero]

On Fri, May 16, 2008 at 10:35 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> Download PsTools, and then say "pslist -m emacs".

Or, if you're using Windows XP Pro or a Server edition, you can simply do

   tasklist /fi "imagename eq emacs*"

 Juanma

Re: Memory leak

[Eli Zaretskii]

> Date: Fri, 16 May 2008 11:03:30 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>, emacs-devel@gnu.org
> 
>    tasklist /fi "imagename eq emacs*"

You need to use emacs.exe as the IMAGENAME, otherwise it will not
work.

(Btw, I think this command is present on XP Home Edition as well.)

Re: Memory leak

[Juanma Barranquero]

On Fri, May 16, 2008 at 11:25 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> You need to use emacs.exe as the IMAGENAME, otherwise it will not
> work.

The wildcard I used in my example works:

C:\> tasklist /fi "imagename eq emacs*"

Image Name                   PID Session Name     Session#    Mem Usage
========================= ====== ================ ======== ============
emacs.exe                   2448 Console                 0     23.528 K

> (Btw, I think this command is present on XP Home Edition as well.)

Ah. I read somewhere that I didn't, but I haven't checked it. In any
case, it can be downloaded off several websites.

   Juanma

Re: Memory leak

[Eli Zaretskii]

> Date: Fri, 16 May 2008 11:29:45 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: lennart.borgman@gmail.com, emacs-devel@gnu.org
> 
> On Fri, May 16, 2008 at 11:25 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > You need to use emacs.exe as the IMAGENAME, otherwise it will not
> > work.
> 
> The wildcard I used in my example works:
> 
> C:\> tasklist /fi "imagename eq emacs*"

Sorry, I missed that little star.

Re: Memory leak

[Juanma Barranquero]

On Fri, May 16, 2008 at 11:29 AM, Juanma Barranquero <lekktu@gmail.com> wrote:

> Ah. I read somewhere that I didn't, but I haven't checked it. In any

Funny. s/I didn't/it didn't/, of course.

 Juanma

Re: Memory leak

[Stephen J. Turnbull]

Juanma Barranquero writes:
 > On Fri, May 16, 2008 at 11:29 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
 > 
 > > Ah. I read somewhere that I didn't, but I haven't checked it. In any
 > 
 > Funny. s/I didn't/it didn't/, of course.

I think that deserves "Typo of the Decade" award!

[OT, silly] Re: Memory leak

[Thomas Lord]

Stephen J. Turnbull wrote:
> Juanma Barranquero writes:
>  > On Fri, May 16, 2008 at 11:29 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>  > 
>  > > Ah. I read somewhere that I didn't, but I haven't checked it. In any
>  > 
>  > Funny. s/I didn't/it didn't/, of course.
>
> I think that deserves "Typo of the Decade" award!
>
>
>
>   


Typo?   Nah, it's the animist "I".  :-)

-t

Re: Memory leak

[David Robinow]

On Fri, May 16, 2008 at 5:25 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Fri, 16 May 2008 11:03:30 +0200
>> From: "Juanma Barranquero" <lekktu@gmail.com>
>> Cc: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>, emacs-devel@gnu.org
>>
>>    tasklist /fi "imagename eq emacs*"
> You need to use emacs.exe as the IMAGENAME, otherwise it will not
> work.
> (Btw, I think this command is present on XP Home Edition as well.)

Apparently not. I have one of each. Works on XP Pro. Not on XP Home.
But thanks, Juanma. I didn't know about that command.

Re: Memory leak

[Lennart Borgman (gmail)]

David Robinow wrote:
> On Fri, May 16, 2008 at 5:25 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> Date: Fri, 16 May 2008 11:03:30 +0200
>>> From: "Juanma Barranquero" <lekktu@gmail.com>
>>> Cc: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>, emacs-devel@gnu.org
>>>
>>>    tasklist /fi "imagename eq emacs*"
>> You need to use emacs.exe as the IMAGENAME, otherwise it will not
>> work.
>> (Btw, I think this command is present on XP Home Edition as well.)
> 
> Apparently not. I have one of each. Works on XP Pro. Not on XP Home.
> But thanks, Juanma. I didn't know about that command.

Thanks Juanma and Eli.

Though while doing things like this I find it more convenient to just do

   tasklist | findstr emacs >> memory-leak.tmp

Re: Memory leak

[Eli Zaretskii]

> Date: Fri, 16 May 2008 16:41:39 -0400
> From: "David Robinow" <drobinow@gmail.com>
> Cc: "Juanma Barranquero" <lekktu@gmail.com>, lennart.borgman@gmail.com, 
> 	emacs-devel@gnu.org
> 
> > (Btw, I think this command is present on XP Home Edition as well.)
> 
> Apparently not. I have one of each. Works on XP Pro. Not on XP Home.

It does on mine, but maybe it came with one of the optional toolkits I
downloaded from the MS site.

Re: Memory leak

[Eli Zaretskii]

> Date: Fri, 16 May 2008 02:25:50 +0200
> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>, emacs-devel@gnu.org
> 
> I can see a memory leak on w32 of around 240 K in the above operation 
> with a build from 2008-03-01

With today's CVS on w32, the memory grows by ~900KB with each new
frame I create, and that memory is not released when the frame and its
buffer are deleted.

Re: Memory leak

[Jason Rumney]

Eli Zaretskii wrote:
> With today's CVS on w32, the memory grows by ~900KB with each new
> frame I create, and that memory is not released when the frame and its
> buffer are deleted.
>   


If the leak is coming from the font cache, then I wouldn't expect it to 
be released when the frame is deleted, as the font cache is shared 
between all frames on w32 (and I think between all frames on the same 
display on X). Stefan is seeing 10x the increase as you are on Windows, 
which could be explained by xft loading fonts into Emacs's process 
space, while on Windows the font itself stays in system owned memory, 
and only Emacs's internal info that it associates with the font 
contributes to the growth. That also points to fonts being responsible 
(though circumstantial).

A while ago I noticed w32font_open being called multiple times for the 
same font, so I suspect that we may be overlooking the font cache in 
places and reopening fonts when we could be reusing an existing font.

There is another similar problem when we get text extents - each redraw 
cycle can call w32font_text_extents a number of times for the same 
glyph_string (from compute_glyph_string_overhangs, left_overwriting and 
right_overwriting in quick succession in draw_glyphs, with nested 
compute_overhangs_and_x within some of the if statements that cause 
extra calls for preceding and following glyph_strings if there is 
overhang, which there often is with antialising enabled). Perhaps this 
operation is cheap with the X font backends, but on Windows it is 
expensive (even with caching of metrics), and I suspect this is a major 
cause of the slowdown reported by Windows users.

Re: Memory leak

[Kenichi Handa]

In article <482D8332.5030908@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> If the leak is coming from the font cache, then I wouldn't expect it to 
> be released when the frame is deleted, as the font cache is shared 
> between all frames on w32 (and I think between all frames on the same 
> display on X). Stefan is seeing 10x the increase as you are on Windows, 
> which could be explained by xft loading fonts into Emacs's process 
> space, while on Windows the font itself stays in system owned memory, 
> and only Emacs's internal info that it associates with the font 
> contributes to the growth. That also points to fonts being responsible 
> (though circumstantial).

I've just installed a fix in font-cache handling.  Now, even
when I create many new frames, the memory usage doesn't
increase that much.  Though it still increases several
handreds K-bytes each time a new frame is created, and it is
not reduced when I delete the frames.  So, I think there
still be a problem somewhere.

FYI, when I compile Emacs without gtk, the amount of
(non-freed) memory increase becomes smaller.

> There is another similar problem when we get text extents - each redraw 
> cycle can call w32font_text_extents a number of times for the same 
> glyph_string (from compute_glyph_string_overhangs, left_overwriting and 
> right_overwriting in quick succession in draw_glyphs, with nested 
> compute_overhangs_and_x within some of the if statements that cause 
> extra calls for preceding and following glyph_strings if there is 
> overhang, which there often is with antialising enabled). Perhaps this 
> operation is cheap with the X font backends, but on Windows it is 
> expensive (even with caching of metrics), and I suspect this is a major 
> cause of the slowdown reported by Windows users.

I remember that I modified x_draw_glyph_string for such a
font-backend as xft that can't do foreground-only drawing
because of antialiasing, which increases the count of
overhang-checking.  Perhaps we should study the code of
draw_glyphs and x_draw_glyph_string and find a solution.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Memory leak

[YAMAMOTO Mitsuharu]

>>>>> On Mon, 19 May 2008 17:05:42 +0900, Kenichi Handa <handa@m17n.org> said:

> I've just installed a fix in font-cache handling.

It seems like the `free_entity' entry in `struct font_driver' no
longer makes sense because you tried to remove the memory leaks by
avoiding Lisp_Save_Value in font entities instead of calling
free_entity at some proper timings.  Is that right?

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: Memory leak

[Kenichi Handa]

In article <wlzlqmu1gw.wl%mituharu@math.s.chiba-u.ac.jp>, YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

>>>>>> On Mon, 19 May 2008 17:05:42 +0900, Kenichi Handa <handa@m17n.org> said:
> > I've just installed a fix in font-cache handling.

> It seems like the `free_entity' entry in `struct font_driver' no
> longer makes sense because you tried to remove the memory leaks by
> avoiding Lisp_Save_Value in font entities instead of calling
> free_entity at some proper timings.  Is that right?

Not exactly.  When I deleted ftfont_free_entity, it was in
my mind to change also ftfont_list to avoid using
Lisp_Save_Value, but I just forgot to implement that change.

The free_entity callback is still called in font_clear_cache
if a font-driver provides that function.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Memory leak

[YAMAMOTO Mitsuharu]

>>>>> On Mon, 19 May 2008 20:13:15 +0900, Kenichi Handa <handa@m17n.org> said:

>> It seems like the `free_entity' entry in `struct font_driver' no
>> longer makes sense because you tried to remove the memory leaks by
>> avoiding Lisp_Save_Value in font entities instead of calling
>> free_entity at some proper timings.  Is that right?

> Not exactly.  When I deleted ftfont_free_entity, it was in my mind
> to change also ftfont_list to avoid using Lisp_Save_Value, but I
> just forgot to implement that change.

> The free_entity callback is still called in font_clear_cache if a
> font-driver provides that function.

So, `free_entity' is supposed to be still valid.  Then I think
the entities deleted in `font_delete_unmatched' need to be freed
as they are not kept in the cache.

Also, there are some suspected leaks in font-related code:

 * `names' in xfont_list_pattern.
 * `charset' in ftfont_spec_pattern when it is created from
   Vscript_representative_chars.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: Memory leak

[Kenichi Handa]

In article <wl4p8ua4ip.wl%mituharu@math.s.chiba-u.ac.jp>, YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

> So, `free_entity' is supposed to be still valid.  Then I think
> the entities deleted in `font_delete_unmatched' need to be freed
> as they are not kept in the cache.

> Also, there are some suspected leaks in font-related code:

>  * `names' in xfont_list_pattern.
>  * `charset' in ftfont_spec_pattern when it is created from
>    Vscript_representative_chars.

Thank you for finding these bugs.  I fixed them.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Memory leak

[David Hansen]

On Mon, 19 May 2008 17:05:42 +0900 Kenichi Handa wrote:

> I've just installed a fix in font-cache handling.  Now, even
> when I create many new frames, the memory usage doesn't
> increase that much.  Though it still increases several
> handreds K-bytes each time a new frame is created, and it is
> not reduced when I delete the frames.  So, I think there
> still be a problem somewhere.

I used to have two frames open in my emacs session, never create or
destroy any other frames.  Memory usage climbs to > 500M here after some
hours of usage.  I doubt it has much to do with frame creation.

Will update right now and report back if your changes help.

David

Re: Memory leak

[David Hansen]

On Mon, 19 May 2008 15:34:20 +0200 David Hansen wrote:

> Will update right now and report back if your changes help.

At least not a that excessive one.  Usage is now at about 70M which is
pretty normal for my emacs usage.

But: - Italics isn't working anymore
     - The Gnus *summary* buffer flickers …
     - … as does looking at some parts of the UTF-8-demo.txt[1] (seems
       to correlate with the display of line drawing characters) …
     - … which is also incredible slooooow (scrolling and cursor
       movements on the timescale of several seconds on my 1.8GHz AMD
       box)
     - Combining characters is broken (at least the example in above
       demo file).

David

Footnotes: 
[1]  http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt

calculation of text extents [Re: Memory leak]

[Kenichi Handa]

In article <482D8332.5030908@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> A while ago I noticed w32font_open being called multiple times for the 
> same font, so I suspect that we may be overlooking the font cache in 
> places and reopening fonts when we could be reusing an existing font.

With the recent fix of font-cache handling, I think the
number of calling `open' callback of a font-backend for the
same font is reduced.  Can you confirm that?

> There is another similar problem when we get text extents - each redraw 
> cycle can call w32font_text_extents a number of times for the same 
> glyph_string (from compute_glyph_string_overhangs, left_overwriting and 
> right_overwriting in quick succession in draw_glyphs, with nested 
> compute_overhangs_and_x within some of the if statements that cause 
> extra calls for preceding and following glyph_strings if there is 
> overhang, which there often is with antialising enabled). Perhaps this 
> operation is cheap with the X font backends, but on Windows it is 
> expensive (even with caching of metrics), and I suspect this is a major 
> cause of the slowdown reported by Windows users.

It seems that we can reduce the number of calling
`text_extents' callback by recording lbearing and rbearing
in struct glyph; which grows the size of that structure from
the current 32-byte to 36-byte.  I'm not sure how much that
change impacts the memory usage of Emacs, but isn't it worth
trying?

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Memory leak

[Stefan Monnier]

> Just to rule out something obvious (pure space overflow):
>   M-: (garbage-collect) RET
> does not return nil, does it?

No.


        Stefan

Re: Memory leak

[YAMAMOTO Mitsuharu]

>>>>> On Thu, 15 May 2008 18:50:29 -0400, Stefan Monnier <monnier@IRO.UMontreal.CA> said:

> We have a pretty serious memory leak.  My Emacs session recently
> died with a memory full situation (the process was around 500MB
> VSZ).  So I took a look around and it seem that every time I create
> a new frame, my Emacs session grows by around 10MB and deleting the
> frame does not recover this memory.  This memory is not accounted
> for by the counts returned by `garbage-collect'.

One thing I'm concerned about is the removal of `ftfont_free_entity'.
Actually, the `leaks' command on Mac OS X shows that several resources
allocated at ftfont_pattern_entity are leaking.

And some minor nitpicking: In xftfont_open, font->pixel_size and
font->driver are set twice (and to different values for
font->pixel_size).

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: Memory leak

[Evil Boris]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> We have a pretty serious memory leak.
> My Emacs session recently died with a memory full situation (the
> process was around 500MB VSZ).  

I do not know if this is related, but sometime after January (around
Unicode merge) I noticed that, after running for a while (have not
been able to notice when precisely this happens, but seems to require
having at least one X frame, sometimes happens within a few minutes of
starting), Emacs starts occupying 190-250Mb of memory.  Since I
usually edit smallish files (<50-100K) and not too many of them (<10,
generally), the only candidate for "eating space" is my RMAIL file,
which is around 30Mb.

Any ideas?   Any obvious place to check what memory is being used
(ancient Solaris 2.7, GCC 3.1)?

Tried on many different CVS versions, including the most current one.

--Boris

PS. Was about to contradict myself---the currently running copy is
179M and has no X frame, but I now recall creating an X frame
yesterday, and then destroying it.

Re: Memory leak

[Stephen J. Turnbull]

Evil Boris writes:

 > I do not know if this is related, but sometime after January (around
 > Unicode merge) I noticed that, after running for a while (have not
 > been able to notice when precisely this happens, but seems to require
 > having at least one X frame, sometimes happens within a few minutes of
 > starting), Emacs starts occupying 190-250Mb of memory.

FWIW, XEmacs has some hard-to-pin-down memory leaks, too, one or more
of which seems to be related to Xft/fontconfig.

These packages are coded in the classic X11 style, meaning that the
burden is entirely on the client code to clean things up.  There *is*
refcounting and things like that, but for internal use of the Xft
code.

I have no idea what the memory use characteristics of underlying
rendering engines such as FreeType might be, so this has been really
hard to track down.

Re: Memory leak

[Eric Hanchrow]

Well, here's a dumb idea: has anyone tried running Emacs under valgrind?
I dimly recall that tool being remarkably useful for finding memory
leaks.
-- 
Governing the U.S. is like playing 200 simultaneous chess
matches (while whiny columnists second-guess every move on every
board).
        -- Nicholas Kristof, New York Times columnist

Re: Memory leak

[Jan Djärv]

Eric Hanchrow skrev:
> Well, here's a dumb idea: has anyone tried running Emacs under valgrind?
> I dimly recall that tool being remarkably useful for finding memory
> leaks.

Valgrind just craches when run on Emacs.  Emacs memory handling is too much 
for valgrind.  It is the undump think that it doesn't understand.

	Jan D.

Re: Memory leak

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> Eric Hanchrow skrev:
>> Well, here's a dumb idea: has anyone tried running Emacs under valgrind?
>> I dimly recall that tool being remarkably useful for finding memory
>> leaks.
>
> Valgrind just craches when run on Emacs.  Emacs memory handling is too
> much for valgrind.  It is the undump think that it doesn't understand.

What about running valgrind on temacs then?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Memory leak

[Jan Djärv]

David Kastrup skrev:
> Jan Djärv <jan.h.d@swipnet.se> writes:
> 
>> Eric Hanchrow skrev:
>>> Well, here's a dumb idea: has anyone tried running Emacs under valgrind?
>>> I dimly recall that tool being remarkably useful for finding memory
>>> leaks.
>> Valgrind just craches when run on Emacs.  Emacs memory handling is too
>> much for valgrind.  It is the undump think that it doesn't understand.
> 
> What about running valgrind on temacs then?
> 

That should work, I haven't tried it.

	Jan D.

Re: Memory leak

[Werner LEMBERG]

> Valgrind just craches when run on Emacs.  Emacs memory handling is too
> much for valgrind.  It is the undump think that it doesn't understand.

What about contacting the valgrind guys directly for help?  I think
that Emacs would get *huge* benefits if checked with it.


    Werner

Re: Memory leak

[Jan Djärv]

Werner LEMBERG skrev:
>> Valgrind just craches when run on Emacs.  Emacs memory handling is too
>> much for valgrind.  It is the undump think that it doesn't understand.
> 
> What about contacting the valgrind guys directly for help?  I think
> that Emacs would get *huge* benefits if checked with it.
> 
> 

If my memory is correct, someone already did that.  I must search the mail 
archives.  I think the Emacs way of doing things was too special for the 
valgrind guys to devote time to.  I actually agree with that.  From what I 
recall, it was the saving and restoring of malloc state during dump and undump 
that was the main problem.  Valgrind has no way of correctly capture this 
state and begin tracking from there.

	Jan D.

Re: Memory leak

[Stephen J. Turnbull]

Jan Djärv writes:

 > Valgrind just craches when run on Emacs.

Portable dumper wins here: XEmacs at least and maybe SXEmacs valgrind
perfectly well.

You should also be able to valgrind temacs.

For more info, you could post to xemacs-beta@xemacs.org (nonmember
posts moderated) and the relevant sxemacs.org list (www.sxemacs.org
for references, dunno about mod policy).

Re: Memory leak

[Jan Djärv]

Stephen J. Turnbull skrev:
> Jan Djärv writes:
> 
>  > Valgrind just craches when run on Emacs.
> 
> Portable dumper wins here: XEmacs at least and maybe SXEmacs valgrind
> perfectly well.

Yes.  But it is not a small project.

> 
> You should also be able to valgrind temacs.
> 

That is a small project :-)

	Jan D.

Re: Memory leak

[Stephen J. Turnbull]

Jan Djärv writes:
 > 
 > 
 > Stephen J. Turnbull skrev:
 > > Jan Djärv writes:
 > > 
 > >  > Valgrind just craches when run on Emacs.
 > > 
 > > Portable dumper wins here: XEmacs at least and maybe SXEmacs valgrind
 > > perfectly well.
 > 
 > Yes.  But it is not a small project.

Um, IIRC it's a done project, though.  Somebody's got a patch in the
queue, no?

I don't know if valgrind is a big enough win to push that into the
mainline now, though.

Re: Memory leak

[Stefan Monnier]

> Um, IIRC it's a done project, though.  Somebody's got a patch in the
> queue, no?

I've heard rumors that someone has worked on it at some point, but
that's a far cry from a "done project", sadly.  For one, I still haven't
seen any patch for it.  And if there's such a patch, I wouldn't be
surprised if it bitrotted significantly.


        Stefan

Re: Memory leak

[Evil Boris]

"Evil Boris" <evilborisnet@netscape.net> writes:

> I do not know if this is related, but sometime after January (around
> Unicode merge) I noticed that, after running for a while (have not
> been able to notice when precisely this happens, but seems to require
> having at least one X frame, sometimes happens within a few minutes of
> starting), Emacs starts occupying 190-250Mb of memory.  Since I
> usually edit smallish files (<50-100K) and not too many of them (<10,
> generally), the only candidate for "eating space" is my RMAIL file,
> which is around 30Mb.

I just checked, with a recent version from CVS [GNU Emacs 23.0.60.9
(sparc-sun-solaris2.7, X toolkit) of 2008-05-21 on ziggy], with no X
frames ever opened, playing almost exclusively with a few small files,
with the sole exception of my 30Mb mailbox, top shows:

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
 1754 ...        1  58    0  174M  172M sleep   4:59  0.00% emacs

174Mb again?  Any ideas?  Any code I can run to try to see what the
memory is being being used for?

--Boris

memory leak

[Drew Adams]

Dunno if this helps at all - I haven't been able to track it down.

I get warnings and then emergency messages saying that Emacs is running out of
memory, after just a few minutes of use. If I don't exit when I first get the
emergency messages, then I end up needing to kill Emacs from outside.

GNU Emacs 23.0.60.1 (i386-mingw-nt5.1.2600)
 of 2008-05-29 on LENNART-69DE564

[It also takes a long time for Emacs 23 to start up. With emacs -Q, it takes
15-30 seconds. But sometimes it starts immediately. I don't get why it's not
regular, but that seems to be the behavior.]

Re: memory leak

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

> Dunno if this helps at all - I haven't been able to track it down.
>
> I get warnings and then emergency messages saying that Emacs is
> running out of memory, after just a few minutes of use. If I don't
> exit when I first get the emergency messages, then I end up needing to
> kill Emacs from outside.

That does not sound like a "memory leak", but the like the mother of all
memory leaks.  Also known as pure space overflow.

What does

M-: (garbage-collect) RET

return?  If nil, bootstrap another Emacs and watch for pure space
overflow messages.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Memory leak

[Nick Roberts]

I'm joining the thread a bit late but anyway:

Instead of Valgrind, which has been shown to be unsuitable several times, how
about using Coverity to identify the source of the memory leak(s)?  

Ben Chelf the CTO for Coverity said "we have set up a framework internally to
continually scan open source projects" and I have the impression that they
scan the most recent Emacs source, or at least can.

Is anyone still signed up to access their reports?  If not, is it worth
approaching them again?

-- 
Nick                                           http://www.inet.net.nz/~nickrob

memory leak

[Robert J. Chassell]

Can anyone else reproduce a memory leak in the current CVS GNU/Emacs?

Debian GNU/Linux, testing, updated this morning
one root console running, one user console running an X windows session

X windows
Enlightenment window manager

Four instances of 
today's GNU Emacs CVS snapshot, Fri, 2006 Apr 28  09:59 UTC
GNU Emacs 22.0.50.64 (i686-pc-linux-gnu, GTK+ Version 2.6.8)
started with

     /usr/local/src/emacs/src/emacs -Q -D

No other instances of Emacs.  During the time of the measurement, I
did nothing with the computer.

These instances of GNU Emacs appear to consume RAM at a rate of 124 kb/min.

How I did it: after instantiating four plain vanilla instances of GNU
Emacs from today's CVS, I recorded the total active memory and then
did that again 10.5 minutes later.

The total active memory comes from `free, and is `used' minus the sum
of `shared, buffers, and cached.  (There was no change in swap size.)

(This is on the usual computer I work with, so it is really painful
getting rid of the instances I use, generally three as a plain user
and two as root; I tend to keep those running for days.)

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: memory leak

[Andreas Schwab]

"Robert J. Chassell" <bob@rattlesnake.com> writes:

> These instances of GNU Emacs appear to consume RAM at a rate of 124 kb/min.

I don't see that on ia64 with fvwm2.  The instance is running for more
than ten days without any apparent leak.  Neither do I see this on ppc64
with kwin, where Emacs is now running for about 5 hours.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: memory leak

[Robert J. Chassell]

   Robert J. Chassell wrote

   > These instances of GNU Emacs appear to consume RAM at a rate of
   > 124 kb/min.

Andreas Schwab wrote

   I don't see that on ia64 with fvwm2.  The instance is running for
   more than ten days without any apparent leak.

Thanks!  I am running an i686 machine.

    Pentium III (Coppermine)
    CPU Speed 846.87 MHz

    gcc --version
    gcc (GCC) 4.0.3 (Debian 4.0.3-1)

I did everything again, just to make sure.  I got rid of Firefox, and
with no Firefox and no instances of Emacs, my memory use (both RAM and
swap) decreased over ten minutes.  With Emacs (-Q -D, my doing
nothing), but no Firefox, RAM use increased (but no change in swap).

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: memory leak

[Luc Teirlinck]

Robert Chassell wrote:

   Can anyone else reproduce a memory leak in the current CVS GNU/Emacs?

Not me.

First of all, I guess that you checked that building Emacs did not
overflow pure storage.  Doing so will definitely produce the results
you describe.  You can check whether the variable pure-space-overflow
is really non-nil or whether (garbage-collect) returns a non-nil value
(it should).

If that is not the problem, I definitely know a way to produce
something that looks very much like a memory leak, although it is not.
All you have to do is have a large number of large files open that
require a lot of fontification, with global-font-lock-mode enabled.
If you have ten or so files the size of lisp/Changelog open, then
after you completely quit using Emacs, the memory consumed by the
Emacs process can easily grow by more than 35 percent, or 8M or more,
over a period spanning an hour or more on a fast machine.  Except for
the apparent "memory leak" there also is an apparent rogue CPU usage.
There is no real memory leak however.  After an hour or more, when all
buffers are completely fontified, both CPU usage and "memory leak"
completely stop.

   These instances of GNU Emacs appear to consume RAM at a rate of 124 kb/min.

I got at least that in my experiments with opening many large files.
But the memory consumption eventually stops, maybe after many hours on
a slow machine.

I can not reproduce anything that even mimics a memory leak without
_both_ global-font-lock-mode being enabled _and_ a large number of
large files requiring heavy fontification being open.

Sincerely,

Luc.

Re: memory leak

[Robert J. Chassell]

This may be it:

    ... after you completely quit using Emacs, the memory consumed by
    the Emacs process can easily grow by more than 35 percent, or 8M
    or more,

My normal instances of Emacs do have a large number of large files
open at once, but the numbers I reported are for when they are killed.

How does memory get consumed after Emacs processes are killed? 

What should I look for after successfully running
`save-buffers-kill-emacs' on all instances of Emacs?

No pure storage overflow:

    You can check whether the variable pure-space-overflow
    is really non-nil or whether (garbage-collect) returns a non-nil value
    (it should).

In both an instance of Emacs started with -Q -D and an instance of my
normal Emacs the variable pure-space-overflow is nil and
(garbage-collect) returns a non-nil value.

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: memory leak

[Luc Teirlinck]

Robert Chassell wrote:

   My normal instances of Emacs do have a large number of large files
   open at once, but the numbers I reported are for when they are killed.

   How does memory get consumed after Emacs processes are killed? 

For me, no appreciable amount of extra memory seems to get consumed
with `emacs -Q -nbc' before Emacs is killed, but I do not have
anything special like desktop enabled.  After Emacs is killed, all of
Emacs' memory should be given back to the OS rather quickly, which
happens for me whenever I kill Emacs.

   What should I look for after successfully running
   `save-buffers-kill-emacs' on all instances of Emacs?

This beats me.  Maybe some other memory consuming process being
launched and surviving the killing of Emacs.  Are you running top with
processes ordered by memory consumption?

Sincerely,

Luc.

Re: memory leak

[Robert J. Chassell]

      What should I look for after successfully running
      `save-buffers-kill-emacs' on all instances of Emacs?

   This beats me.  Maybe some other memory consuming process being
   launched and surviving the killing of Emacs.  Are you running top with
   processes ordered by memory consumption?

No, I was not running top when I did the measurements, but I was
running a lot of processes.  At one point, I thought that Firefox was
the culprit, but it appears not.  I don't know where to go from here,
excepting onto another machine.  At least, no one else sees an Emacs
problem, which is good.

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: memory leak

[Luc Teirlinck]

Robert Chassell wrote:

   No, I was not running top when I did the measurements, but I was
   running a lot of processes.  At one point, I thought that Firefox was
   the culprit, but it appears not.  I don't know where to go from here,
   excepting onto another machine.

Is there any reason why you can not identify the culprit by running
top, ordered by memory consumption, whenever the problem occurs?  I
guess you can also use ps if you prefer static output.

You can just start top and press `h' to see how to specify sort order
(it varies with the top implementation).  The actual options you have
to give to ps to sort by size or resident memory also vary from
implementation to implementation, but they are in the man pages.

Sincerely,

Luc.

Re: memory leak

[Robert J. Chassell]

   Is there any reason why you can not identify the culprit by running
   top, ordered by memory consumption, whenever the problem occurs?  I
   guess you can also use ps if you prefer static output.

That is what caused me to think first that Firefox was the culprit.
My version does not appear to reuse memory properly when a page is
renewed ... but that is a different issue.  I got an increase when I
did not have Firefox running, but did have Emacs running (that is to
say, running instances of `emacs -Q -D' with no files open and my
doing nothing).  I got no increase when neither Firefox nor Emacs were
running.  That is why I became suspicious of Emacs.
 
Anyhow, you have a good point.  I am becoming less and less inclined
to worry about Emacs and more inclined to think I was fooled by one of
the new processes I do not know anything about.

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: memory leak

[Richard Stallman]

When he wrote

	... after you completely quit using Emacs, the memory consumed by
	the Emacs process can easily grow by more than 35 percent, or 8M
	or more,

I think he really meant "when you stop typing and leave Emacs
completely idle".

Re: memory leak

[Luc Teirlinck]

Richard Stallman wrote:

   When he wrote

	   ... after you completely quit using Emacs, the memory consumed by
	   the Emacs process can easily grow by more than 35 percent, or 8M
	   or more,

   I think he really meant "when you stop typing and leave Emacs
   completely idle".

Yes, that is what I meant.  When Emacs becomes idle and stays idle,
jit-lock and stealth fontification kick in.  If you have a lot of
large buffers that require a lot of fontification, this mimics a
memory leak, as well as rogue CPU usage.  But both CPU usage and
memory increase eventually stop.  However, due to the way jit-lock
implements its "niceness", the process can go on for more than an
hour, even on a relatively fast machine, as long as you have enough
large buffers open.

Sincerely,

Luc.