Memory Usage On Emacs 24 Lucid Linux:

Memory Usage On Emacs 24 Lucid Linux:

[T. V. Raman]

Just did a git pull -a to see if nmy memory woes on the 16-core
workstation (HP Z600) were a thing of the past, alas No.

Here are some numbers that I hope are useful:

Running Emacs 23.2 on that machine, with my default .emacs
loaded, RSS  is about 235MB VSS about 430MB. This is an Emacs
session that has been up for about a week.

Emacs 24 built from today's git snapshot:

Starting Emacs with the same .emacs loaded:

0. All memory on the machine goes away to Emacs (started with 9GB
of 12GB free) -- as reported by free.

1. After running a garbage collect in that emacs (before GC, it  
was basically  too slow to use)
Running proced showed an RSS of 7gb and a VSS of approx 8GB.

I ran M-x memory-usage and here are the numbers.

Garbage collection stats:
((833982 . 321192) (57906 . 5) (116 . 491) 4504763 1000738985 (1539 . 520) (13100 . 7508) (171211 . 33823))

 =>	13343712+5139072 bytes in cons cells
	2779488+240 bytes in symbols
	4640+19640 bytes in markers
	24624+8320 bytes in floats
	733600+420448 bytes in intervals
	5478752+1082336 bytes in string headers
	4504763 bytes of string chars
	4504763 bytes of vector slots

Total bytes in lisp objects: 1034278620 (live 1027608564, dead 6670056)

Buffer ralloc memory usage:
18 buffers
113342 bytes total (4948 in gaps)
      Size	Gap	Name

     45482	0	*Compile-Log*
     43734	1852	*Proced*
      7714	771	.diary
      6742	111	*Messages*
      2039	204	 *g scratch*
       596	59	*Calendar*
       594	59	*shell*
       568	1570	*Buffer Details*
       362	36	*Warnings*
       322	32	 *completion-save-buffer*
       191	20	*scratch*
        81	20	*Shell Command Output*
        42	20	 *code-conversion-work*
        12	20	 *Echo Area 0*
         9	20	 *dtk-scratch-buffer* 
         0	20	 *Minibuf-1*
         0	20	 *Minibuf-0*
         0	20	 *Echo Area 1*


-- 

-- 

-- 

Re: Memory Usage On Emacs 24 Lucid Linux:

[Stefan Monnier]

> Just did a git pull -a to see if nmy memory woes on the 16-core
> workstation (HP Z600) were a thing of the past, alas No.

> Here are some numbers that I hope are useful:

> Running Emacs 23.2 on that machine, with my default .emacs
> loaded, RSS  is about 235MB VSS about 430MB. This is an Emacs
> session that has been up for about a week.

> Emacs 24 built from today's git snapshot:

> Starting Emacs with the same .emacs loaded:

> 0. All memory on the machine goes away to Emacs (started with 9GB
> of 12GB free) -- as reported by free.

> 1. After running a garbage collect in that emacs (before GC, it  
> was basically  too slow to use)
> Running proced showed an RSS of 7gb and a VSS of approx 8GB.

> I ran M-x memory-usage and here are the numbers.

> Garbage collection stats:
> ((833982 . 321192) (57906 . 5) (116 . 491) 4504763 1000738985 (1539 . 520) (13100 . 7508) (171211 . 33823))

>  =>	13343712+5139072 bytes in cons cells
> 	2779488+240 bytes in symbols
> 	4640+19640 bytes in markers
> 	24624+8320 bytes in floats
> 	733600+420448 bytes in intervals
> 	5478752+1082336 bytes in string headers
> 	4504763 bytes of string chars
> 	4504763 bytes of vector slots

Hmm... there's a bug in the above output: it should say 1000738985 bytes
of vector slots.  That a whole 1G of vector slots.  Where could these
come from?

Can you reproduce the above output with just "emacs; M-x memory-usage"
or do you need to do something more?  Can you try to figure out which
part of your setup triggers this behavior?


        Stefan

Re: Memory Usage On Emacs 24 Lucid Linux:

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Mon, 12 Mar 2012 16:33:23 -0400
> Cc: emacs-devel@gnu.org
> 
> Hmm... there's a bug in the above output: it should say 1000738985 bytes
> of vector slots.  That a whole 1G of vector slots.  Where could these
> come from?

Putting a breakpoint on allocate_vectorlike and looking at the Lisp
backtrace should probably bring the answer to that pretty fast.

Re: Memory Usage On Emacs 24 Lucid Linux:

[Eli Zaretskii]

> Date: Mon, 12 Mar 2012 13:13:55 -0700
> From: "T. V. Raman" <tv.raman.tv@gmail.com>
> 
> Emacs 24 built from today's git snapshot:
> 
> Starting Emacs with the same .emacs loaded:
> 
> 0. All memory on the machine goes away to Emacs (started with 9GB
> of 12GB free) -- as reported by free.
> 
> 1. After running a garbage collect in that emacs (before GC, it  
> was basically  too slow to use)
> Running proced showed an RSS of 7gb and a VSS of approx 8GB.

Could you please file a formal bug report?  One advantage of that is
that it would record all the pertinent data about your configuration,
packages that are loaded, etc.

Thanks.

Re: Memory Usage On Emacs 24 Lucid what?:

[Richard Stallman]

What is "Lucid Linux"?  Is it a GNU/Linux distro?

--
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Memory Usage On Emacs 24 Lucid what?:

[Simon Leinen]

[-- Attachment #1: Type: text/plain, Size: 285 bytes --]

On Tue, Mar 13, 2012 at 04:51, Richard Stallman <rms@gnu.org> wrote:

> What is "Lucid Linux"?  Is it a GNU/Linux distro?


Probably a typo :-) "Lucid Lynx" is a release of the Ubuntu GNU/Linux
distribution.  This release is also called "10.04 LTS" (for long-time
support).
-- 
Simon.

[-- Attachment #2: Type: text/html, Size: 571 bytes --]

Memory Usage On Emacs 24 Lucid Linux:

[T.V. Raman]

Hi Stephane,

I started an emacs -q -- loaded memory-usage.el -- and produced
memory-usage report -- attached below.

It looks normal, as do the numbers reported by free  no leak.

My emacs world is admittedly complicated. If I load emacspeak and
things start talking -- so emacs -q -l <emacspeak-setup> then I
do see the memory numbers blowing up.

If it is any help, emacspeak uses advice *heavily*.
On the other hand, I am running emacs 24 successfuly on a 64 bit
laptop and a 32-bit home desktop machine.

Garbage collection stats:
((51337 . 9533) (14771 . 0) (73 . 107) 76102 325117 (67 . 81) (180 .
54) (3028 . 4598))

 =>	821392+152528 bytes in cons cells
	709008+0 bytes in symbols
	2920+4280 bytes in markers
	1072+1296 bytes in floats
	10080+3024 bytes in intervals
	96896+147136 bytes in string headers
	76102 bytes of string chars
	76102 bytes of vector slots

Total bytes in lisp objects: 2350851 (live 2042587, dead 308264)

Buffer ralloc memory usage:
10 buffers
8865 bytes total (2382 in gaps)
      Size	Gap	Name

      4632	463	memory-usage.el
       778	77	*GNU Emacs*
       521	1597	*Buffer Details*
       329	33	*Messages*
       191	20	*scratch*
        82	20	 *Echo Area 0*
        42	20	 *code-conversion-work*
         0	20	 *Minibuf-1*
         0	20	 *Minibuf-0*
         0	20	 *Echo Area 1*



On 3/12/12, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Just did a git pull -a to see if nmy memory woes on the 16-core
>> workstation (HP Z600) were a thing of the past, alas No.
>
>> Here are some numbers that I hope are useful:
>
>> Running Emacs 23.2 on that machine, with my default .emacs
>> loaded, RSS  is about 235MB VSS about 430MB. This is an Emacs
>> session that has been up for about a week.
>
>> Emacs 24 built from today's git snapshot:
>
>> Starting Emacs with the same .emacs loaded:
>
>> 0. All memory on the machine goes away to Emacs (started with 9GB
>> of 12GB free) -- as reported by free.
>
>> 1. After running a garbage collect in that emacs (before GC, it
>> was basically  too slow to use)
>> Running proced showed an RSS of 7gb and a VSS of approx 8GB.
>
>> I ran M-x memory-usage and here are the numbers.
>
>> Garbage collection stats:
>> ((833982 . 321192) (57906 . 5) (116 . 491) 4504763 1000738985 (1539 . 520)
>> (13100 . 7508) (171211 . 33823))
>
>>  =>	13343712+5139072 bytes in cons cells
>> 	2779488+240 bytes in symbols
>> 	4640+19640 bytes in markers
>> 	24624+8320 bytes in floats
>> 	733600+420448 bytes in intervals
>> 	5478752+1082336 bytes in string headers
>> 	4504763 bytes of string chars
>> 	4504763 bytes of vector slots
>
> Hmm... there's a bug in the above output: it should say 1000738985 bytes
> of vector slots.  That a whole 1G of vector slots.  Where could these
> come from?
>
> Can you reproduce the above output with just "emacs; M-x memory-usage"
> or do you need to do something more?  Can you try to figure out which
> part of your setup triggers this behavior?
>
>
>         Stefan
>

Memory Usage On Emacs 24 Lucid Linux:

[T.V. Raman]

Stephane et al

I am also possibly able to trigger the problem before I load
emacspeak;

After running emacs -q -- and then M-x shell, I see all of the
memory on the machine getting shewed up -- at least as reported
by free.

I'm attaching outupt of memory-usage from both instances so they
are in the same message;  see what you think.

The memory usage numbers with M-x shell dont look as big as the
ones with just emacs -q -- but they do look bigger.

Plain: memory usage in emacs -q

Garbage collection stats:
((51337 . 9533) (14771 . 0) (73 . 107) 76102 325117 (67 . 81) (180 .
54) (3028 . 4598))

 =>	821392+152528 bytes in cons cells
	709008+0 bytes in symbols
	2920+4280 bytes in markers
	1072+1296 bytes in floats
	10080+3024 bytes in intervals
	96896+147136 bytes in string headers
	76102 bytes of string chars
	76102 bytes of vector slots

Total bytes in lisp objects: 2350851 (live 2042587, dead 308264)

Buffer ralloc memory usage:
10 buffers
8865 bytes total (2382 in gaps)
      Size	Gap	Name

      4632	463	memory-usage.el
       778	77	*GNU Emacs*
       521	1597	*Buffer Details*
       329	33	*Messages*
       191	20	*scratch*
        82	20	 *Echo Area 0*
        42	20	 *code-conversion-work*
         0	20	 *Minibuf-1*
         0	20	 *Minibuf-0*
         0	20	 *Echo Area 1*
Shell: emacs -q M-x shell M-x memory-usage

Garbage collection stats:
((54977 . 10205) (15045 . 0) (90 . 121) 417724 1000327014 (67 . 191)
(117 . 161) (25648 . 1043))

 =>	879632+163280 bytes in cons cells
	722160+0 bytes in symbols
	3600+4840 bytes in markers
	1072+3056 bytes in floats
	6552+9016 bytes in intervals
	820736+33376 bytes in string headers
	417724 bytes of string chars
	417724 bytes of vector slots

Total bytes in lisp objects: 1003392058 (live 1003178490, dead 213568)

Buffer ralloc memory usage:
10 buffers
4360 bytes total (1967 in gaps)
      Size	Gap	Name

       827	82	*GNU Emacs*
       537	1590	*Buffer Details*
       489	49	*shell*
       345	34	*Messages*
       191	20	*scratch*
        81	20	 *Echo Area 0*
        15	20	 *code-conversion-work*
         0	20	 *Minibuf-1*
         0	20	 *Minibuf-0*
         0	20	 *Echo Area 1*

Full: My Full Emacs environment, everything loaded

Garbage collection stats:
((833982 . 321192) (57906 . 5) (116 . 491) 4504763 1000738985 (1539 .
520) (13100 . 7508) (171211 . 33823))

 =>	13343712+5139072 bytes in cons cells
	2779488+240 bytes in symbols
	4640+19640 bytes in markers
	24624+8320 bytes in floats
	733600+420448 bytes in intervals
	5478752+1082336 bytes in string headers
	4504763 bytes of string chars
	4504763 bytes of vector slots

Total bytes in lisp objects: 1034278620 (live 1027608564, dead 6670056)

Buffer ralloc memory usage:
18 buffers
113342 bytes total (4948 in gaps)
      Size	Gap	Name

     45482	0	*Compile-Log*
     43734	1852	*Proced*
      7714	771	.diary
      6742	111	*Messages*
      2039	204	 *g scratch*
       596	59	*Calendar*
       594	59	*shell*
       568	1570	*Buffer Details*
       362	36	*Warnings*
       322	32	 *completion-save-buffer*
       191	20	*scratch*
        81	20	*Shell Command Output*
        42	20	 *code-conversion-work*
        12	20	 *Echo Area 0*
         9	20	 *dtk-scratch-buffer*
         0	20	 *Minibuf-1*
         0	20	 *Minibuf-0*
         0	20	 *Echo Area 1*


On 3/12/12, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Just did a git pull -a to see if nmy memory woes on the 16-core
>> workstation (HP Z600) were a thing of the past, alas No.
>
>> Here are some numbers that I hope are useful:
>
>> Running Emacs 23.2 on that machine, with my default .emacs
>> loaded, RSS  is about 235MB VSS about 430MB. This is an Emacs
>> session that has been up for about a week.
>
>> Emacs 24 built from today's git snapshot:
>
>> Starting Emacs with the same .emacs loaded:
>
>> 0. All memory on the machine goes away to Emacs (started with 9GB
>> of 12GB free) -- as reported by free.
>
>> 1. After running a garbage collect in that emacs (before GC, it
>> was basically  too slow to use)
>> Running proced showed an RSS of 7gb and a VSS of approx 8GB.
>
>> I ran M-x memory-usage and here are the numbers.
>
>> Garbage collection stats:
>> ((833982 . 321192) (57906 . 5) (116 . 491) 4504763 1000738985 (1539 . 520)
>> (13100 . 7508) (171211 . 33823))
>
>>  =>	13343712+5139072 bytes in cons cells
>> 	2779488+240 bytes in symbols
>> 	4640+19640 bytes in markers
>> 	24624+8320 bytes in floats
>> 	733600+420448 bytes in intervals
>> 	5478752+1082336 bytes in string headers
>> 	4504763 bytes of string chars
>> 	4504763 bytes of vector slots
>
> Hmm... there's a bug in the above output: it should say 1000738985 bytes
> of vector slots.  That a whole 1G of vector slots.  Where could these
> come from?
>
> Can you reproduce the above output with just "emacs; M-x memory-usage"
> or do you need to do something more?  Can you try to figure out which
> part of your setup triggers this behavior?
>
>
>         Stefan
>

Re: Memory Usage On Emacs 24 Lucid Linux:

[Stefan Monnier]

> After running emacs -q -- and then M-x shell, I see all of the
> memory on the machine getting shewed up -- at least as reported
> by free.

Aha!

> Shell: emacs -q M-x shell M-x memory-usage

> Garbage collection stats:
> ((54977 . 10205) (15045 . 0) (90 . 121) 417724 1000327014 (67 . 191)
> (117 . 161) (25648 . 1043))

>  =>	879632+163280 bytes in cons cells
> 	722160+0 bytes in symbols
> 	3600+4840 bytes in markers
> 	1072+3056 bytes in floats
> 	6552+9016 bytes in intervals
> 	820736+33376 bytes in string headers
> 	417724 bytes of string chars
> 	417724 bytes of vector slots

> Total bytes in lisp objects: 1003392058 (live 1003178490, dead 213568)

OK, that's very helpful, it narrows it down a good deal.
Now, all we have to figure out is how on earth does "M-x shell" create
so many (or such large) vectors!
Could it be you have $HISTSIZE set to a really large number?


        Stefan

Memory Usage On Emacs 24 Lucid Linux:

[T.V. Raman]

I'll check tomorrow when I get to work, but you may be correct; I
do remember setting up a histsize hack a long time ago so that
Ididn't have to remember long, complex commandline  utterances.

This would also explain why this is biting only on the office
workstation and not at either home or on the laptop.
-- 
Best Regards,
--raman

-- 
Best Regards,
--raman


On 3/13/12, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> After running emacs -q -- and then M-x shell, I see all of the
>> memory on the machine getting shewed up -- at least as reported
>> by free.
>
> Aha!
>
>> Shell: emacs -q M-x shell M-x memory-usage
>
>> Garbage collection stats:
>> ((54977 . 10205) (15045 . 0) (90 . 121) 417724 1000327014 (67 . 191)
>> (117 . 161) (25648 . 1043))
>
>>  =>	879632+163280 bytes in cons cells
>> 	722160+0 bytes in symbols
>> 	3600+4840 bytes in markers
>> 	1072+3056 bytes in floats
>> 	6552+9016 bytes in intervals
>> 	820736+33376 bytes in string headers
>> 	417724 bytes of string chars
>> 	417724 bytes of vector slots
>
>> Total bytes in lisp objects: 1003392058 (live 1003178490, dead 213568)
>
> OK, that's very helpful, it narrows it down a good deal.
> Now, all we have to figure out is how on earth does "M-x shell" create
> so many (or such large) vectors!
> Could it be you have $HISTSIZE set to a really large number?
>
>
>         Stefan
>

Memory Usage On Emacs 24 Lucid Linux:

[T.V. Raman]

Stefan,

You were spot on --- I had HISTSIZE  set to infinity.

This is a setting I have had for the last 6+ years at work, I'll
append the shell configuration bits I was using to essentially
have an infinitely long history. It worked correctly until about
May 2011, which is when Emacs 24 built from git/bzr started
displaying the memory leak behavior we have been discussing.

I've removed these from my shell environemnt, and emacs 24 is
running happily with my full emacs environment loaded.

Below, I'll append the shell setup that caused the problem --
suspect just taking out histsize would have fixed it, but I'm
showing the set up here because I believe it would be valuable
for emacs folks who understand the bash interaction to come up
with a simpler, cleaner way of doing this.


export command_oriented_history=1
export history_control ignoredups
shopt -s histappend
export HISTSIZE=999999999
homeraman
home/raman
export TVR_SHELL_LOG=/home/raman/.jotlog/shell.log




On 3/13/12, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> After running emacs -q -- and then M-x shell, I see all of the
>> memory on the machine getting shewed up -- at least as reported
>> by free.
>
> Aha!
>
>> Shell: emacs -q M-x shell M-x memory-usage
>
>> Garbage collection stats:
>> ((54977 . 10205) (15045 . 0) (90 . 121) 417724 1000327014 (67 . 191)
>> (117 . 161) (25648 . 1043))
>
>>  =>	879632+163280 bytes in cons cells
>> 	722160+0 bytes in symbols
>> 	3600+4840 bytes in markers
>> 	1072+3056 bytes in floats
>> 	6552+9016 bytes in intervals
>> 	820736+33376 bytes in string headers
>> 	417724 bytes of string chars
>> 	417724 bytes of vector slots
>
>> Total bytes in lisp objects: 1003392058 (live 1003178490, dead 213568)
>
> OK, that's very helpful, it narrows it down a good deal.
> Now, all we have to figure out is how on earth does "M-x shell" create
> so many (or such large) vectors!
> Could it be you have $HISTSIZE set to a really large number?
>
>
>         Stefan
>

Re: Memory Usage On Emacs 24 Lucid Linux:

[Stefan Monnier]

> You were spot on --- I had HISTSIZE  set to infinity.

Great.

> have an infinitely long history. It worked correctly until about
> May 2011, which is when Emacs 24 built from git/bzr started
> displaying the memory leak behavior we have been discussing.

1G entries, leads to 4GB or 8GB array, so I'd guess that before May
2011, Emacs still computed those sizes using `int' (32bit) or something
like that.
Or it's simply when shell.el started to support HISTSIZE.

> I've removed these from my shell environemnt, and Emacs 24 is
> running happily with my full Emacs environment loaded.

So the right fix might be for shell.el to allocate its history-ring
more lazily.


        Stefan

Re: Memory Usage On Emacs 24 Lucid Linux:

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've removed these from my shell environemnt, and Emacs 24 is
>> running happily with my full Emacs environment loaded.
>
> So the right fix might be for shell.el to allocate its history-ring
> more lazily.

Implemented in trunk.  (Though if HISTSIZE is huge and the user really
does have gigabytes of shell history stored on disk, Emacs still loads
it all; that can't really be helped).