Emacs bzr memory footprint

Emacs bzr memory footprint

[Carsten Mattner]

I've been wondering whether it's normal for Emacs to start with
10 or 20 megs and stay at ~100 megs after opening and closing
multiple buffers.

I do use ido and a couple other common modes/libs for various
features or programming languages. Nothing fancy, really.
I can try to test with no custom config, but won't be able to use
it for real work, just letting it idle.

Is this in the normal footprint or way out of place when
Emacs sits there idling with not a single buffer open.

Emacs versions in use are Linux with X but no toolkit and
Cocoa Emacs, both from bzr.

Does Emacs have some kind of memory reclaim or gc mechanism?

Speaking about memory, I've always wondered why mmap is listed
as not enabled by configure. Is it something which is experimental
or detrimental?

Thanks,

Carsten

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> Date: Thu, 13 Oct 2011 11:24:25 +0200
> From: Carsten Mattner <carstenmattner@googlemail.com>
> 
> I've been wondering whether it's normal for Emacs to start with
> 10 or 20 megs and stay at ~100 megs after opening and closing
> multiple buffers.

It depends on what you mean by "opening and closing".  My Emacs
session is running for 19 hours and its current footprint is 187MB,
but it has 273 buffers of various sizes.  This is on a 32-bit machine.

So please show a minimal sequence of commands that bring you from 20MB
up to 100MB.  Please also tell what platform is this on (64-bit hosts
are expected to use more memory than 32-bit hosts).

> Does Emacs have some kind of memory reclaim or gc mechanism?

Of course, Emacs includes garbage collector.  It runs automatically,
but you can also invoke it manually with "M-x garbage-collect RET".

> Speaking about memory, I've always wondered why mmap is listed
> as not enabled by configure.

I think mmap is always used if available and workable.  The configure
script includes a test for it, it announces itself as "checking for
working mmap".  Does this test fail for you?  If so, look in
config.log for the reasons.

Re: Emacs bzr memory footprint

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> I think mmap is always used if available and workable.

No.

use_mmap_for_buffers=no
case "$opsys" in
  cygwin|freebsd|irix6-5) use_mmap_for_buffers=yes ;;
esac

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Carsten Mattner <carstenmattner@googlemail.com>,  emacs-devel@gnu.org
> Date: Thu, 13 Oct 2011 12:33:15 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think mmap is always used if available and workable.
> 
> No.
> 
> use_mmap_for_buffers=no
> case "$opsys" in
>   cygwin|freebsd|irix6-5) use_mmap_for_buffers=yes ;;
> esac

Thanks, I confused the OP's question with whether malloc uses mmap.

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Thu, Oct 13, 2011 at 12:46 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: Carsten Mattner <carstenmattner@googlemail.com>,  emacs-devel@gnu.org
>> Date: Thu, 13 Oct 2011 12:33:15 +0200
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> > I think mmap is always used if available and workable.
>>
>> No.
>>
>> use_mmap_for_buffers=no
>> case "$opsys" in
>>   cygwin|freebsd|irix6-5) use_mmap_for_buffers=yes ;;
>> esac
>
> Thanks, I confused the OP's question with whether malloc uses mmap.

As I remember that configure result from Linux and glibc uses
mmap, from what I remember, being disabled makes sense.

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Thu, Oct 13, 2011 at 12:01 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 13 Oct 2011 11:24:25 +0200
>> From: Carsten Mattner <carstenmattner@googlemail.com>
>>
>> I've been wondering whether it's normal for Emacs to start with
>> 10 or 20 megs and stay at ~100 megs after opening and closing
>> multiple buffers.
>
> It depends on what you mean by "opening and closing".  My Emacs
> session is running for 19 hours and its current footprint is 187MB,
> but it has 273 buffers of various sizes.  This is on a 32-bit machine.

Also 32-bit machine here.

"opening and closing" means loading some source file (C, Erlang,
Text, Haskell, M4, something else) and maybe erc-tls. Then after nuking
all buffers and emacs using minimum 100 megs I wondered why no
gc kicked in to reduce it by at least 30 megs (random number).

Are there known leaks with erlang-mode or haskell-mode?
Can there be leaks in major modes or other addons like
ido extensions or custom themes?

> So please show a minimal sequence of commands that bring you from 20MB
> up to 100MB.  Please also tell what platform is this on (64-bit hosts
> are expected to use more memory than 32-bit hosts).

Darwin 64bit and Linux 32bit

>> Does Emacs have some kind of memory reclaim or gc mechanism?
>
> Of course, Emacs includes garbage collector.  It runs automatically,
> but you can also invoke it manually with "M-x garbage-collect RET".

I believe I tried that to no effect. Will retry and take notes.

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> Date: Thu, 13 Oct 2011 15:08:28 +0200
> From: Carsten Mattner <carstenmattner@googlemail.com>
> Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs-devel@gnu.org
> 
> On Thu, Oct 13, 2011 at 12:46 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Cc: Carsten Mattner <carstenmattner@googlemail.com>,  emacs-devel@gnu.org
> >> Date: Thu, 13 Oct 2011 12:33:15 +0200
> >>
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >>
> >> > I think mmap is always used if available and workable.
> >>
> >> No.
> >>
> >> use_mmap_for_buffers=no
> >> case "$opsys" in
> >>   cygwin|freebsd|irix6-5) use_mmap_for_buffers=yes ;;
> >> esac
> >
> > Thanks, I confused the OP's question with whether malloc uses mmap.
> 
> As I remember that configure result from Linux and glibc uses
> mmap, from what I remember, being disabled makes sense.

Configuring on GNU/Linux shows that mmap is not used directly for
buffers, which is consistent with the above.

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> Date: Thu, 13 Oct 2011 15:13:55 +0200
> From: Carsten Mattner <carstenmattner@googlemail.com>
> Cc: emacs-devel@gnu.org
> 
> "opening and closing" means loading some source file (C, Erlang,
> Text, Haskell, M4, something else) and maybe erc-tls. Then after nuking
> all buffers and emacs using minimum 100 megs I wondered why no
> gc kicked in to reduce it by at least 30 megs (random number).

Don't forget that visiting all these files invoked the corresponding
major and minor modes, which were then loaded by Emacs, and that
consumes memory.  Emacs does not unload packages automatically, so
that memory continues to be in use even if you kill the buffers.

In addition, there are internal buffers and data structures, such as
load-history, which grow for as long as your session is up, and are
not shrunk when you kill buffers.

I don't know if several tens of MBs can be explained by this, but some
of them surely can.

> Are there known leaks with erlang-mode or haskell-mode?

There can be no memory leak on the Lisp side.  Any leaks are always
bugs in Emacs, not in any Lisp code that it runs.

Re: Emacs bzr memory footprint

[Dave Abrahams]

on Thu Oct 13 2011, Eli Zaretskii <eliz-AT-gnu.org> wrote:

>> Are there known leaks with erlang-mode or haskell-mode?
>
> There can be no memory leak on the Lisp side.  Any leaks are always
> bugs in Emacs, not in any Lisp code that it runs.

I'm not an expert on Emacs internals, but I think that depends on
whether they are memory leaks or "memory leaks," the latter being when
references are held longer than necessary.  Also, presumably, until
something forces a garbage collection, even unreferenced memory may
appear to be "in use."

Please do correct me if I'm mistaken.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Emacs bzr memory footprint

[Stefan Monnier]

> There can be no memory leak on the Lisp side.  Any leaks are always
> bugs in Emacs, not in any Lisp code that it runs.

That's not quite true: indeed in a GC'd language, there cannot be leaks
in the C sense of "objects still allocated while we don't even have
a reference to them any more".  But leaks in the sense of "objects still
kept around even know we will never ever use them again" can
still happen.


        Stefan


PS: BTW, since you use erlang-mode: could you get someone from the
    erlang-mode developers to contact me in order to include it in Emacs
    (I've tried several times but got no answer).

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: Dave Abrahams <dave@boostpro.com>
> Date: Thu, 13 Oct 2011 10:10:19 -0400
> 
> 
> on Thu Oct 13 2011, Eli Zaretskii <eliz-AT-gnu.org> wrote:
> 
> >> Are there known leaks with erlang-mode or haskell-mode?
> >
> > There can be no memory leak on the Lisp side.  Any leaks are always
> > bugs in Emacs, not in any Lisp code that it runs.
> 
> I'm not an expert on Emacs internals, but I think that depends on
> whether they are memory leaks or "memory leaks," the latter being when
> references are held longer than necessary.  Also, presumably, until
> something forces a garbage collection, even unreferenced memory may
> appear to be "in use."

Well, that's true, but then this is not a "memory leak", it's
incorrect use of memory.

Actually, the more I think about this, the more I wonder how would a
Lisp program "keep around" unneeded objects.  Maybe if it interns many
symbols, or defines a lot of global variables.  Or maybe I'm missing
something.  Let's put it this way: I think a Lisp program needs to
work very hard to produce this kind of "leaks".

Re: Emacs bzr memory footprint

[Dave Abrahams]

on Thu Oct 13 2011, Eli Zaretskii <eliz-AT-gnu.org> wrote:

>> From: Dave Abrahams <dave@boostpro.com>
>
>> Date: Thu, 13 Oct 2011 10:10:19 -0400
>> 
>> 
>> on Thu Oct 13 2011, Eli Zaretskii <eliz-AT-gnu.org> wrote:
>> 
>> >> Are there known leaks with erlang-mode or haskell-mode?
>> >
>> > There can be no memory leak on the Lisp side.  Any leaks are always
>> > bugs in Emacs, not in any Lisp code that it runs.
>> 
>> I'm not an expert on Emacs internals, but I think that depends on
>> whether they are memory leaks or "memory leaks," the latter being when
>> references are held longer than necessary.  Also, presumably, until
>> something forces a garbage collection, even unreferenced memory may
>> appear to be "in use."
>
> Well, that's true, but then this is not a "memory leak", it's
> incorrect use of memory.
>
> Actually, the more I think about this, the more I wonder how would a
> Lisp program "keep around" unneeded objects.  Maybe if it interns many
> symbols, or defines a lot of global variables.  Or maybe I'm missing
> something.  Let's put it this way: I think a Lisp program needs to
> work very hard to produce this kind of "leaks".

Typical examples arise with caches of various sorts.  This is a
well-known phenomenon in the Java world, and there's nothing particular
to lisp or Java that should make lisp any different.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Emacs bzr memory footprint

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
>> As I remember that configure result from Linux and glibc uses
>> mmap, from what I remember, being disabled makes sense.
>
> Configuring on GNU/Linux shows that mmap is not used directly for
> buffers, which is consistent with the above.

... and deleting big buffers definitely frees lots of memory, so
something's working...

but I note that even after killing most buffers and doing a gc, my emacs
(running for roughly 8 hours) still takes 341 MB... :[

-miles

-- 
=====
(^o^;
(()))
*This is the cute octopus virus, please copy it into your sig so it can spread.

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Cc: Carsten Mattner <carstenmattner@googlemail.com>,  schwab@linux-m68k.org,  emacs-devel@gnu.org
> Date: Fri, 14 Oct 2011 00:22:03 +0900
> 
> but I note that even after killing most buffers and doing a gc, my emacs
> (running for roughly 8 hours) still takes 341 MB... :[

What happens if you map over load-history, unloading each feature
mentioned there?

Re: Emacs bzr memory footprint

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> In addition, there are internal buffers and data structures, such as
> load-history, which grow for as long as your session is up, and are
> not shrunk when you kill buffers.
>
> I don't know if several tens of MBs can be explained by this, but some
> of them surely can.

Don't forget fonts; visiting HELLO will bump your memory usage up by
about 10MB.  And loading the unicode name database (e.g. for C-x 8 RET
completion) will bump it up by another 10MB.

Re: Emacs bzr memory footprint

[Stefan Monnier]

>> but I note that even after killing most buffers and doing a gc, my emacs
>> (running for roughly 8 hours) still takes 341 MB... :[
> What happens if you map over load-history, unloading each feature
> mentioned there?

My guess is that checking the return value of `garbage-collect' would
probably indicate (without unloading anything) that such unloads cannot
recover anywhere near 300MB because the Lisp heap is much smaller
(i.e. the data comes from elsewhere).


        Stefan

Re: Emacs bzr memory footprint

[Stephen J. Turnbull]

Miles Bader writes:

 > but I note that even after killing most buffers and doing a gc, my emacs
 > (running for roughly 8 hours) still takes 341 MB... :[

XEmacs is quite a bit more bloated than that, but I can't find most of
it in XEmacs's memory reporting statistics.  My groundless suspicion
is that a lot of it is in caches, images, whatnot in X and other
dependency libraries.  Except that now that Emacs is showing some bloat
too, the ground isn't moving under my feet quite so hard. :-)

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Thu, Oct 13, 2011 at 4:22 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> PS: BTW, since you use erlang-mode: could you get someone from the
>    erlang-mode developers to contact me in order to include it in Emacs
>    (I've tried several times but got no answer).

I must admit that I prefer to have haskell-mode and erlang-mode
be not part of Emacs' repo.
I find it favorable to have an up-to-date custom-managed or
package.el managed version.
For example with cc-mode, if Alan didn't agree to update it
soon in Emacs bzr, I was about to build and use a copy
in ~/.emacs.d.

In summary: we shouldn't try to add more big packages to Emacs,
especially not such which see more frequent releases than Emacs.
We have package support for a reason, don't we?

I will contact the erlang-mode maintainer, just to let him know,
though.

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Thu, Oct 13, 2011 at 6:46 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
> Miles Bader writes:
>
>  > but I note that even after killing most buffers and doing a gc, my emacs
>  > (running for roughly 8 hours) still takes 341 MB... :[
>
> XEmacs is quite a bit more bloated than that, but I can't find most of
> it in XEmacs's memory reporting statistics.  My groundless suspicion
> is that a lot of it is in caches, images, whatnot in X and other
> dependency libraries.  Except that now that Emacs is showing some bloat
> too, the ground isn't moving under my feet quite so hard. :-)

Would that be not loaded if I started it up as 'emacs -nw'?
Might be worth a try to see how much less it is, wouldn't it?
Actually, I'd be interested to know what you guys see with that
in comparison.

I've never got used to -nw because there's always either
some key-bindings or color support discrepancy compared
to X or Cocoa clients.

The Linux X Emacs I configure as follows:
/configure --without-selinux --without-tiff --without-sound \
      --without-dbus --without-gconf --without-gsettings --without-xft \
      --without-gsettings --with-x-toolkit=no

On OSX:
./configure --with-ns --without-gnutls

Re: Emacs bzr memory footprint

[Stefan Monnier]

> I must admit that I prefer to have haskell-mode and erlang-mode
> be not part of Emacs' repo.

I'm not planning to add erlang-mode to Emacs, but to the GNU ELPA.


        Stefan

Re: Emacs bzr memory footprint

[Richard Riley]

Carsten Mattner <carstenmattner@googlemail.com> writes:

> On Thu, Oct 13, 2011 at 4:22 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>> PS: BTW, since you use erlang-mode: could you get someone from the
>>    erlang-mode developers to contact me in order to include it in Emacs
>>    (I've tried several times but got no answer).
>
> I must admit that I prefer to have haskell-mode and erlang-mode
> be not part of Emacs' repo.
> I find it favorable to have an up-to-date custom-managed or
> package.el managed version.
> For example with cc-mode, if Alan didn't agree to update it
> soon in Emacs bzr, I was about to build and use a copy
> in ~/.emacs.d.
>
> In summary: we shouldn't try to add more big packages to Emacs,
> especially not such which see more frequent releases than Emacs.
> We have package support for a reason, don't we?
>
> I will contact the erlang-mode maintainer, just to let him know,
> though.

Whats to stop you installing the latest versions regardless of whether
it ships basic emacs? Far better for commonly used modes to ship with
emacs for fallback in case the newer versions are hosed for that version
of emacs for whatever reason.

Re: Emacs bzr memory footprint

[Dmitry Antipov]

I tried the following simple scenario with latest bzr trunk:

1. Run ./src/emacs, then eval (length features), which gives 88. VSS/RSS is 195436/18656 Kb.
2. Run ./src/emacs `find . -name '*.el' -o -name '*.[ch]'`, wait, eval (length features) =>
    100, (length (buffer-list)) => 1990. VSS/RSS is 309264/132656 kb.
3. After (mapcar 'kill-buffer (buffer-list)), VSS/RSS is 273620/97132 kb.

I believe 97132 - 18656 = 78476 kb of physical memory is too much for 12 Lisp libraries.
Where is the rest? Or, what's wrong with this scenario?

Dmitry

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> Date: Thu, 13 Oct 2011 21:33:54 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> 
> I tried the following simple scenario with latest bzr trunk:
> 
> 1. Run ./src/emacs, then eval (length features), which gives 88. VSS/RSS is 195436/18656 Kb.
> 2. Run ./src/emacs `find . -name '*.el' -o -name '*.[ch]'`, wait, eval (length features) =>
>     100, (length (buffer-list)) => 1990. VSS/RSS is 309264/132656 kb.
> 3. After (mapcar 'kill-buffer (buffer-list)), VSS/RSS is 273620/97132 kb.
> 
> I believe 97132 - 18656 = 78476 kb of physical memory is too much for 12 Lisp libraries.
> Where is the rest? Or, what's wrong with this scenario?

What is the return value of (garbage-collect), before and after these
actions?

Re: Emacs bzr memory footprint

[chad]

On Oct 13, 2011, at 6:13 AM, Carsten Mattner wrote:
> "opening and closing" means loading some source file (C, Erlang,
> Text, Haskell, M4, something else) and maybe erc-tls. Then after nuking
> all buffers and emacs using minimum 100 megs I wondered why no
> gc kicked in to reduce it by at least 30 megs (random number).

A shot in the dark: Emacs' use of gnu-tls libraries is relatively new; 
can you reproduce the behavior without invoking the tls libraries?  
For example, try an emacs without using erc-tls, (and without gnus 
or w3, if you use those)?

If you use recentf, then you can mine 'recentf-list (or ~/.recentf) for 
a list of files to load that probably just involve programming modes.

*Chad

Re: Emacs bzr memory footprint

[Dan Nicolaescu]

Carsten Mattner <carstenmattner@googlemail.com> writes:

> I've been wondering whether it's normal for Emacs to start with
> 10 or 20 megs and stay at ~100 megs after opening and closing
> multiple buffers.
>
> I do use ido and a couple other common modes/libs for various
> features or programming languages. Nothing fancy, really.
> I can try to test with no custom config, but won't be able to use
> it for real work, just letting it idle.
>
> Is this in the normal footprint or way out of place when
> Emacs sits there idling with not a single buffer open.
>
> Emacs versions in use are Linux with X but no toolkit and
> Cocoa Emacs, both from bzr.
>
> Does Emacs have some kind of memory reclaim or gc mechanism?

Memory can only be return back to the system in chunks (pages usually)
and only when the complete chunk becomes free. Because how the
allocation works it is possible to have on the same chunk data that is
free (like a buffer that has been killed) and data that is not: history,
lisp functions and data, so it is never completely free. 

As a data point we are not doing too bad, this emacs from work:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                 
19053 dann      16   0  133m 129m 3864 S  0.0  1.6 177:42.07 emacs                    

has been running for 212 days, 3 hours, 45 minutes, 2 seconds

it has 202 open buffers, the terminal frame count has reached 678 
and the memory usage is still reasonable after that much time of daily
usage .

Re: Emacs bzr memory footprint

[Dmitry Antipov]

On 10/13/2011 10:42 PM, Eli Zaretskii wrote:

>> 1. Run ./src/emacs, then eval (length features), which gives 88. VSS/RSS is 195436/18656 Kb.
>> 2. Run ./src/emacs `find . -name '*.el' -o -name '*.[ch]'`, wait, eval (length features) =>
>>      100, (length (buffer-list)) =>  1990. VSS/RSS is 309264/132656 kb.
>> 3. After (mapcar 'kill-buffer (buffer-list)), VSS/RSS is 273620/97132 kb.
>>
>> I believe 97132 - 18656 = 78476 kb of physical memory is too much for 12 Lisp libraries.
>> Where is the rest? Or, what's wrong with this scenario?
>
> What is the return value of (garbage-collect), before and after these
> actions?

-- conses ----------- symbols ------ markers ----- string chars - vector slots - floats --- intervals ----- strings--------
1) (49153 . 8818)     (14376 . 0)    (36 . 149)    70230          322172         (68 . 225) (36 . 10)       (3241 . 2153)
2) (1007686 . 123595) (37604 . 0)    (4002 . 3546) 984071         1396344        (70 . 122) (126865 . 5201) (24249 . 11060)
3) (84021 . 133132)   (20840 . 7631) (45 . 273)    589623         445180         (70 . 327) (38 . 214)      (15836 . 18822)

Dmitry

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Thu, Oct 13, 2011 at 7:09 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> I must admit that I prefer to have haskell-mode and erlang-mode
>> be not part of Emacs' repo.
>
> I'm not planning to add erlang-mode to Emacs, but to the GNU ELPA.

That sounds good. I can support that :).

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Thu, Oct 13, 2011 at 10:18 PM, chad <yandros@mit.edu> wrote:
>
> On Oct 13, 2011, at 6:13 AM, Carsten Mattner wrote:
>> "opening and closing" means loading some source file (C, Erlang,
>> Text, Haskell, M4, something else) and maybe erc-tls. Then after nuking
>> all buffers and emacs using minimum 100 megs I wondered why no
>> gc kicked in to reduce it by at least 30 megs (random number).
>
> A shot in the dark: Emacs' use of gnu-tls libraries is relatively new;
> can you reproduce the behavior without invoking the tls libraries?
> For example, try an emacs without using erc-tls, (and without gnus
> or w3, if you use those)?

I will try to use irssi instead and see what the difference is.
For that I will disable loading any of the tls libs in init.el.
Also I will try to use 'emacs -nw' for comparison.

> If you use recentf, then you can mine 'recentf-list (or ~/.recentf) for
> a list of files to load that probably just involve programming modes.

I don't use recentf, but I do use ido and idomenu.

Re: Emacs bzr memory footprint

[Carsten Mattner]

For the record, I also use evil, because I can't live without
the quick and efficient editing of vi command mode.
It's a mix of vi and Emacs bindings combining the best
of both worlds.
I like Emacs and its extensibility, but miss efficient editing
of vi's command mode. I'm not particularly in favor of
vi(m) itself.
I mean even when I play around with Haskell's Yi editor
framework I tend to use the vi input configuration.

Should I also try to disable evil?

From what I see in others' memory footprints my experience
doesn't seem uncommon.
Therefore I don't actually believe I have a local issue that's
specific to my config.

Re: Emacs bzr memory footprint

[Carsten Mattner]

I have to correct myself.

The bzr Emacs version I build on OSX is actually 32-bit
as is the 32-bit version on 32-bit Linux.

Re: Emacs bzr memory footprint

[Sven Joachim]

On 2011-10-14 14:18 +0200, Carsten Mattner wrote:

> For the record, I also use evil, because I can't live without
> the quick and efficient editing of vi command mode.
> It's a mix of vi and Emacs bindings combining the best
> of both worlds.
> I like Emacs and its extensibility, but miss efficient editing
> of vi's command mode. I'm not particularly in favor of
> vi(m) itself.
> I mean even when I play around with Haskell's Yi editor
> framework I tend to use the vi input configuration.
>
> Should I also try to disable evil?

Probably won't help much.

> From what I see in others' memory footprints my experience
> doesn't seem uncommon.
> Therefore I don't actually believe I have a local issue that's
> specific to my config.

I'm also experiencing memory leaks.  My current session is at 82M, with
the accumulated size of all buffers less than 1M; visiting files and
then deleting their buffers does not increase memory footprint, but Gnus
usage (visiting groups and articles) slowly but steadily does.

Sven

Re: Emacs bzr memory footprint

[Ted Zlatanov]

On Fri, 14 Oct 2011 17:49:16 +0200 Sven Joachim <svenjoac@gmx.de> wrote: 

SJ> I'm also experiencing memory leaks.  My current session is at 82M, with
SJ> the accumulated size of all buffers less than 1M; visiting files and
SJ> then deleting their buffers does not increase memory footprint, but Gnus
SJ> usage (visiting groups and articles) slowly but steadily does.

It's possible GnuTLS usage is part of the problem.  Do you use it?

Another possibility is libxml, used by the Gnus SHR HTML renderer.  Do
you look at articles with HTML content a lot?

It could also be the way that SHR loads images.  Do those articles have
images?

Could you try *not* to use each possibly problematic feature above and
see if the memory usage is affected?

Thanks
Ted

Re: Emacs bzr memory footprint

[Sven Joachim]

On 2011-10-14 18:30 +0200, Ted Zlatanov wrote:

> On Fri, 14 Oct 2011 17:49:16 +0200 Sven Joachim <svenjoac@gmx.de> wrote: 
>
> SJ> I'm also experiencing memory leaks.  My current session is at 82M, with
> SJ> the accumulated size of all buffers less than 1M; visiting files and
> SJ> then deleting their buffers does not increase memory footprint, but Gnus
> SJ> usage (visiting groups and articles) slowly but steadily does.
>
> It's possible GnuTLS usage is part of the problem.  Do you use it?

Yes.

> Another possibility is libxml, used by the Gnus SHR HTML renderer.  Do
> you look at articles with HTML content a lot?

No, rarely.

> It could also be the way that SHR loads images.  Do those articles have
> images?

Very few have.

> Could you try *not* to use each possibly problematic feature above and
> see if the memory usage is affected?

Is setting starttls-use-gnutls to nil enough to disable the first
feature?  And how do I disable the second and third?

Cheers,
       Sven

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Fri, Oct 14, 2011 at 6:48 PM, Sven Joachim <svenjoac@gmx.de> wrote:
> On 2011-10-14 18:30 +0200, Ted Zlatanov wrote:
>
>> On Fri, 14 Oct 2011 17:49:16 +0200 Sven Joachim <svenjoac@gmx.de> wrote:
>>
>> SJ> I'm also experiencing memory leaks.  My current session is at 82M, with
>> SJ> the accumulated size of all buffers less than 1M; visiting files and
>> SJ> then deleting their buffers does not increase memory footprint, but Gnus
>> SJ> usage (visiting groups and articles) slowly but steadily does.
>>
>> It's possible GnuTLS usage is part of the problem.  Do you use it?
>
> Yes.
>
>> Another possibility is libxml, used by the Gnus SHR HTML renderer.  Do
>> you look at articles with HTML content a lot?
>
> No, rarely.
>
>> It could also be the way that SHR loads images.  Do those articles have
>> images?
>
> Very few have.
>
>> Could you try *not* to use each possibly problematic feature above and
>> see if the memory usage is affected?
>
> Is setting starttls-use-gnutls to nil enough to disable the first
> feature?  And how do I disable the second and third?

I have disabled evil, tls, starttls and erc.
Then I loaded one erlang file, let it sit there for a couple hours,
loaded ~/.emacs.d/init.el. Then killed both buffers, let it sit there
and it's still only at 24.8megs so far. Calling garbage-collect
make the RES go up to 24.9 and goes down to 24.8 soon after.
That's of course unscientific, so no need to tell me that :).

By the way, I'm glad there's a lively discussion around this topic and
that I'm not the only one affected.

Re: Emacs bzr memory footprint

[James Cloos]

>>>>> "CM" == Carsten Mattner <carstenmattner@googlemail.com> writes:

CM> I've been wondering whether it's normal for Emacs to start with
CM> 10 or 20 megs and stay at ~100 megs after opening and closing
CM> multiple buffers.

How many fonts do you have installed?  In how many directories?

fontconfig mmap(2)s every fc cache file, bloating the VM numbers.
But those are read-only, so every app uses the same chunk of ram
for each of those files, just like shared libraries.

The gtk gui pulls in quite a few libraries.  That also will bloat
the VM numbers.

You may also end up with several font files mmap(2)ed by fc.  Again,
read-only.

Look at the output of:

     ;: lsof -p $(pgrep emacs)

It should give you an idea of how many files are mmap(2)ed.

On linux you also can look at /proc/$PID/maps.  The BSDs also may
support something like that.  My emacs maps file has 656 entries.
And that is with the (lighter-weight) athena toolkit.

One third of those mmap(2)ed files are fontconfig cache files.

(Note that, for the typical library, the elf sections -- such as data,
rodata, text, etc -- are separately mmap(2)ed.  So there will be three
to four times as many library lines in maps as libraries loaded.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Emacs bzr memory footprint

[John Wiegley]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I must admit that I prefer to have haskell-mode and erlang-mode be not part
>> of Emacs' repo.

> I'm not planning to add erlang-mode to Emacs, but to the GNU ELPA.

In that same vein, Emacs Chess should go into the GNU ELPA, and not into Emacs
proper.  It's a large package, and not related to the general task of editing
really in any way.  Plus, it depends on pixmaps from XBoard (fortunately, also
a GNU project) which weigh several megabytes by themselves.

John

Re: Emacs bzr memory footprint

[Stefan Monnier]

>>> I must admit that I prefer to have haskell-mode and erlang-mode be not part
>>> of Emacs' repo.
>> I'm not planning to add erlang-mode to Emacs, but to the GNU ELPA.
> In that same vein, Emacs Chess should go into the GNU ELPA, and not into Emacs
> proper.

Yup.  I didn't know we were planning to include Emacs Chess, but yes I'd
be happy to see it included in GNU ELPA,


        Stefan

Re: Emacs bzr memory footprint

[Sven Joachim]

On 2011-10-14 18:48 +0200, Sven Joachim wrote:

> On 2011-10-14 18:30 +0200, Ted Zlatanov wrote:
>
>> On Fri, 14 Oct 2011 17:49:16 +0200 Sven Joachim <svenjoac@gmx.de> wrote: 
>>
>> SJ> I'm also experiencing memory leaks.  My current session is at 82M, with
>> SJ> the accumulated size of all buffers less than 1M; visiting files and
>> SJ> then deleting their buffers does not increase memory footprint, but Gnus
>> SJ> usage (visiting groups and articles) slowly but steadily does.
>>
>> It's possible GnuTLS usage is part of the problem.  Do you use it?
>
> Yes.

I rebuilt Emacs "--without-gnutls", and the memory leaks are gone.  My
current instance, started 11 hours ago, is at 51M, with an ERC session
running and 296 features loaded.  This is about the same memory
footprint as in Emacs 23.

Cheers,
       Sven

Re: Emacs bzr memory footprint

[Juri Linkov]

> On linux you also can look at /proc/$PID/maps.

Not in Emacs, unfortunately, that displays files from the proc filesystem
incompletely, truncated at the position 65536.  Is it possible to fix this?

Re: Emacs bzr memory footprint

[James Cloos]

>>>>> "JL" == Juri Linkov <juri@jurta.org> writes:

JC>> On linux you also can look at /proc/$PID/maps.

JL> Not in Emacs, unfortunately, that displays files from the proc filesystem
JL> incompletely, truncated at the position 65536.  Is it possible to fix this?

Odd.  Something in emacs must do that; nothing else here has that bug.

That said, though, busybox’s vi fails to display /proc/*/maps files at all.

But I cannot find anything else -- editor or otherwise -- which fails to
show the entire maps file.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Emacs bzr memory footprint

[Andreas Schwab]

James Cloos <cloos@jhcloos.com> writes:

>>>>>> "JL" == Juri Linkov <juri@jurta.org> writes:
>
> JC>> On linux you also can look at /proc/$PID/maps.
>
> JL> Not in Emacs, unfortunately, that displays files from the proc filesystem
> JL> incompletely, truncated at the position 65536.  Is it possible to fix this?
>
> Odd.  Something in emacs must do that; nothing else here has that bug.

	  /* The file size returned from stat may be zero, but data
	     may be readable nonetheless, for example when this is a
	     file in the /proc filesystem.  */
	  if (end_offset == 0)
	    end_offset = READ_BUF_SIZE;

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs bzr memory footprint

[Nix]

On 14 Oct 2011, Sven Joachim said:

> On 2011-10-14 18:30 +0200, Ted Zlatanov wrote:
>
>> On Fri, 14 Oct 2011 17:49:16 +0200 Sven Joachim <svenjoac@gmx.de> wrote: 
>>
>> SJ> I'm also experiencing memory leaks.  My current session is at 82M, with
>> SJ> the accumulated size of all buffers less than 1M; visiting files and
>> SJ> then deleting their buffers does not increase memory footprint, but Gnus
>> SJ> usage (visiting groups and articles) slowly but steadily does.
>>
>> It's possible GnuTLS usage is part of the problem.  Do you use it?
>
> Yes.

So do I.

Here are two Emacses (quite old, I'm afraid, bzr 105407, must upgrade):

STIME   RSS    VSZ
Oct07 832348 1127088
Oct07 226916 499588

The first Emacs does little but run Gnus and BBDB, including a bunch of
gnutls-accessed IMAP mailboxes at my employer's (Drew can probably tell
you *exactly* how that works, even if I use Gnus to access them and he
doesn't), but also including nnml and a bunch of ordinary nntp groups on
multiple servers. No images, no multicharset stuff except whatever I run
across on emacs-devel :)

The second Emacs does no Gnus, but does *all* my software development,
my entire day job, and is much more heavily used, with far more active
buffers, including monsters like a TAGS file for the entire Linux
kernel. Yet its memory consumption is far lower. They have identical
configurations otherwise.

(garbage-collect) for the first:

((2118391 . 1218350) (83106 . 0) (80362 . 45727) 87827825 2736712 (7831 . 20893) (94543 . 15062) (461420 . 158890))

For the second:

((1362224 . 894226) (51114 . 79) (17329 . 6290) 9620209 952486 (747 . 4077) (84546 . 5233) (336029 . 181268))

Gnus is clearly driving Emacs much harder than mere cc-mode ever does.
I'm particularly stunned by the number of string-chars in use. It must
have some huge variables defined, probably holding overview data or
something... I'm not sure how large some of those things are: if they're
large structures this might explain most or all of the memory
consumption.

Re: Emacs bzr memory footprint

[John Wiegley]

>>>>> Nix  <nix@esperi.org.uk> writes:

> Gnus is clearly driving Emacs much harder than mere cc-mode ever does.  I'm
> particularly stunned by the number of string-chars in use. It must have some
> huge variables defined, probably holding overview data or something... I'm
> not sure how large some of those things are: if they're large structures
> this might explain most or all of the memory consumption.

Do you using the Gnus Registry?  Which backends do you use?  How many messages
are in mailboxes that you open?

John

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, John Wiegley verbalised:

>>>>>> Nix  <nix@esperi.org.uk> writes:
>
>> Gnus is clearly driving Emacs much harder than mere cc-mode ever does.  I'm
>> particularly stunned by the number of string-chars in use. It must have some
>> huge variables defined, probably holding overview data or something... I'm
>> not sure how large some of those things are: if they're large structures
>> this might explain most or all of the memory consumption.
>
> Do you using the Gnus Registry?

I hadn't noticed it existed, so no (though it looks rather nice and I
shall probably be using it in due course, pushing my memory usage up
even further).

>                                  Which backends do you use?

nnml, nntp, a couple of tiny nndoc, and one, one single huge nnmh group
(containing filtered spam less than six months old).

>                                                              How many messages
> are in mailboxes that you open?

Some of them are quite large: my primary nnml mailbox has 16000 mails in
it, with a 6Mb overview (though only 60-odd are visible). Some of my
nntp groups never expire so are very, very large (100000-odd articles),
but, again, only a few dozen to a few hundred articles will be unread
and visible at any time. (The largest overview for a single group is
46Mb, but if Gnus is reading the entire overview database for an nntp
group in, it's doing something wrong!)

However, this cannot explain the memory consumption, because I check
most of these groups out within a few minutes of starting Emacs, and
memory consumption then is around 300Mb. The rise from then on is
inexorable, though not steady: where the figures for ten hours ago were

STIME   RSS    VSZ
Oct07 832348 1127088
Oct07 226916 499588

now they are

STIME   RSS    VSZ
Oct07 876524 1170572
Oct07 227016 499588

So the coding Emacs has hardly budged, but the newsreading one has
chewed up another 50Mb. It's a good thing this machine has 24Gb RAM :)

pmap shows thwe following sizeable anonymous regions:

0000000001f18000 850596K rw---    [ anon ]
00007fbf8ff29000      4K rw---    [ anon ]
00007fbf98000000    132K rw---    [ anon ]
00007fbf98021000  65404K -----    [ anon ]
00007fbf9f9b4000      4K -----    [ anon ]
00007fbf9f9b5000  32768K rw---    [ anon ]
00007fbfa1f6d000    772K rw---    [ anon ]
00007fbfa22b6000   1260K rw---    [ anon ]
00007fbfa266a000    236K rw---    [ anon ]
00007fbfa2ca1000     76K rw---    [ anon ]
00007fbfa2dc2000    772K rw---    [ anon ]
00007fbfa2e83000     28K rw---    [ anon ]
00007fbfa2ec9000      4K rw---    [ anon ]
00007fbfa2f0e000   1544K rw---    [ anon ]
00007fbfa309b000    168K rw---    [ anon ]

Heap fragmentation might explain this, but most of the big allocations
(e.g. for huge overviews) should be going into separately mmap()ed
regions and getting freed, not into that 850Mb pig of a heap. With luck
it's just one, but luck does not accompany me on trips like this. (I too
spent some time fruitlessly instrumenting XEmacs for signs of the cause
of its huge memory usage and came to the same conclusion as Stephen:
it's not XEmacs, it's the toolkits. The same may be true here, though at
least I'm using Lucid, not Gtk, so we can rule out *that* mountain of
code.)

Also note that XEmacs's huge memory usage was accompanied by a radical
slowdown in GC times that eventually forced a restart if I was to get
anything done. By contrast, this ballooning is not accompanied by any
slowdown in GC: a GC still takes only about 1/5s, barely slower than
when Emacs is freshly started.

Hm.


On second thought, IIRC Gnus allocates some very large lists as part of
its overview management or something: perhaps this is serving to spam
the arena with a huge number of (individually small, thus not
mmap()-allocated) atoms which, when they get freed later, produce a very
sparsely-filled, severely-fragmented heap? If so, perhaps Emacs would
benefit from a simple pool allocator accessed via a new let/setq form or
a new arg to create-buffer, so Gnus could arrange to stuff variables it
knows will be huge, or buffer-local variables of buffers it thinks may
have lots of huge buffer-local vars, into a newly-mmap()ed region?
Unfortunately that means, sigh, using our own malloc() again, which is
probably more painful than useful. I suspect actually proving my
contention first would be a good idea. Not sure how to get the addresses
of Lisp objects from a running Emacs though: gdb, presumably.

-- 
NULL && (void)

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: Nix <nix@esperi.org.uk>
> Emacs: well, why *shouldn't* you pay property taxes on your editor?
> Date: Fri, 21 Oct 2011 01:19:53 +0100
> Cc: emacs-devel@gnu.org
> 
> On second thought, IIRC Gnus allocates some very large lists as part of
> its overview management or something: perhaps this is serving to spam
> the arena with a huge number of (individually small, thus not
> mmap()-allocated) atoms which, when they get freed later, produce a very
> sparsely-filled, severely-fragmented heap? If so, perhaps Emacs would
> benefit from a simple pool allocator accessed via a new let/setq form or
> a new arg to create-buffer, so Gnus could arrange to stuff variables it
> knows will be huge, or buffer-local variables of buffers it thinks may
> have lots of huge buffer-local vars, into a newly-mmap()ed region?
> Unfortunately that means, sigh, using our own malloc() again, which is
> probably more painful than useful. I suspect actually proving my
> contention first would be a good idea. Not sure how to get the addresses
> of Lisp objects from a running Emacs though: gdb, presumably.

Do you see a similar growth of the footprint in Emacs 23.3?  None of
what you are describing seems to be specific to Emacs 24, so it would
be good to compare with the memory usage of Emacs 23.3.

As for memory fragmentation: this is on GNU/Linux, right?  If so, the
memory allocator used by Emacs should prevent any such fragmentation,
at least in the simple scenarios you described.  So I don't think
memory fragmentation can explain what you see.

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Eli Zaretskii outgrape:

> Do you see a similar growth of the footprint in Emacs 23.3?  None of
> what you are describing seems to be specific to Emacs 24, so it would
> be good to compare with the memory usage of Emacs 23.3.

Sorry, I'm afraid I switched from XEmacs to Emacs during the 24.x
development period and jumped straight onto the trunk, so I have no
experience of 23.x. Ripping the lexbinds back out of my .emacs would
be extremely unpleasant, but I'll see if I can manage it :)

> As for memory fragmentation: this is on GNU/Linux, right?

Yes.

>                                                            If so, the
> memory allocator used by Emacs should prevent any such fragmentation,
> at least in the simple scenarios you described.  So I don't think
> memory fragmentation can explain what you see.

What, even if what's happening is massive allocation of conses followed
by allocation of longer-lived stuff followed by freeing of those conses?
Conses are small: there's no way they'll go over the M_MMAP_THRESHOLD.

-- 
NULL && (void)

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: Nix <nix@esperi.org.uk>
> Cc: jwiegley@gmail.com, emacs-devel@gnu.org
> Date: Fri, 21 Oct 2011 13:05:42 +0100
> 
> >                                                            If so, the
> > memory allocator used by Emacs should prevent any such fragmentation,
> > at least in the simple scenarios you described.  So I don't think
> > memory fragmentation can explain what you see.
> 
> What, even if what's happening is massive allocation of conses followed
> by allocation of longer-lived stuff followed by freeing of those conses?

I'm really not an expert on this, but I think the answer is YES.  At
least the GNU/Linux build does not use the gmalloc.c and ralloc.c
modules, AFAIK because the native malloc on GNU/Linux includes the
same or superior functionality.

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Eli Zaretskii said:

>> Date: Fri, 21 Oct 2011 13:05:42 +0100
>> What, even if what's happening is massive allocation of conses followed
>> by allocation of longer-lived stuff followed by freeing of those conses?
>
> I'm really not an expert on this, but I think the answer is YES.  At
> least the GNU/Linux build does not use the gmalloc.c and ralloc.c
> modules, AFAIK because the native malloc on GNU/Linux includes the
> same or superior functionality.

Unfortunately it won't release memory back to the OS in the situation I
described. mmap() is expensive, so only large allocations (by
default >128Kb or >512Kb depending on glibc version and arch) get
mmap()ed. If you want to e.g. release all storage allocated for a buffer
*and its local variables* when that buffer is freed, you need a pool
allocator which can be told to allocate those local variables in a
buffer-specific mmap()ed pool... and glibc malloc has nothing like it.
(I'm thinking here of something like APR's pool allocators, though
obviously not actually those allocators. Something with a similar design
though.)

But this is all speculation. I'm running a debugging patch now courtesy
of Dimitry Antipov: we'll see what it turns up in the next few days (or
even possibly hours).

-- 
NULL && (void)

Re: Emacs bzr memory footprint

[Stefan Monnier]

> However, this cannot explain the memory consumption, because I check
> most of these groups out within a few minutes of starting Emacs, and
> memory consumption then is around 300Mb. The rise from then on is
> inexorable, though not steady: where the figures for ten hours ago were

> STIME   RSS    VSZ
> Oct07 832348 1127088
> Oct07 226916 499588

> now they are

> STIME   RSS    VSZ
> Oct07 876524 1170572
> Oct07 227016 499588

I've just installed memory-usage.el as a package in the `elpa' branch
(you'll soon be able to install it using package.el and in the mean time
you can download it from http://bzr.savannah.gnu.org/lh/emacs/elpa/download/head:/memoryusage.el-20111021130523-o1v7kfuzrat3pcd9-2/memory-usage.el).

It basically looks at the garbage-collect output and the various
buffer's size to provide you with a human-readable description of the
memory in use, from Elisp's point of view.  Don't expect too much of it,
but it can be helpful.

Could you try it out?
If this output does not explain the process's size you're seeing, then
we have a leak in the C code somewhere.  If it does, then we have either
a leak in Elisp, or at least an excessive memory use by some package,
and hopefully we can at least figure out which category of object
is involved.

> Also note that XEmacs's huge memory usage was accompanied by a radical
> slowdown in GC times that eventually forced a restart if I was to get
> anything done. By contrast, this ballooning is not accompanied by any
> slowdown in GC: a GC still takes only about 1/5s, barely slower than
> when Emacs is freshly started.

A fast GC means that there are fairly few Elisp objects, hence most of
the memory is used either by objects not visible to Elisp, or by things
like large strings or large buffers (since the GC doesn't need to scan
the string text or buffer text).

> sparsely-filled, severely-fragmented heap? If so, perhaps Emacs would
> benefit from a simple pool allocator accessed via a new let/setq form or
> a new arg to create-buffer, so Gnus could arrange to stuff variables it
> knows will be huge, or buffer-local variables of buffers it thinks may
> have lots of huge buffer-local vars, into a newly-mmap()ed region?

Indeed Emacs does not provide any way to tell the allocator to colocate
objects into a particular "pool" so they can be freed together.

> Unfortunately that means, sigh, using our own malloc() again, which is
> probably more painful than useful.

Emacs already uses its own malloc for most objects: it uses malloc
directly to allocate vectors (and this is actually something that we
might want to change, because it (indirectly) comes with an O(log n)
cost), but for strings, conses, markers, overlays, and floats it uses
malloc only to get a chunk of memory which it then manages on its own.
`memory-usage' does give you this kind of info when it says
"5743392+864696 bytes in cons cells" which means 5MB of live cons-cells,
and 800KB of cons-cells that are free (i.e. they are in a memory chunk
that Emacs can't return to malloc because that chunk also contains live
cons cells).

BTW, from the GC's point of view (and memory-usage's as well), "vectors"
include a few non-vector objects such as buffers, processes,
hash-tables, and a few more; and "markers" similarly include a few
non-marker objects, mostly overlays.  Oh, and "intervals" are objects
used to store text-properties.

> I suspect actually proving my contention first would be a good
> idea.  Not sure how to get the addresses of Lisp objects from a running
> Emacs though: gdb, presumably.

I'd hack src/alloc.c to export the needed info to Elisp.  But maybe
memory-usage will already give us enough info that this won't
be necessary.


        Stefan

Re: Emacs bzr memory footprint

[Dmitry Antipov]

On 10/21/2011 04:47 PM, Nix wrote:

> Unfortunately it won't release memory back to the OS in the situation I
> described. mmap() is expensive, so only large allocations (by
> default>128Kb or>512Kb depending on glibc version and arch) get
> mmap()ed.

On GNU/Linux system, Emacs tweaks this default with mallopt (M_MMAP_THRESHOLD, 64*1024).

Nevertheless, glibc malloc design assumes that smaller (below threshold above) chunks may
be still allocated with mmap(). Run emacs with:

strace -e signal=none -e trace=mmap,mremap,munmap emacs

and you will see a lot of allocation requests of very different sizes.

On the other side, lisp data is newer mapped, and in some cases this is the main reason
which causes sbrk()'ed heap fragmentation. Don't beat toolkits, don't beat system libraries.
Now I'm observing the case where 4 GCs reduces amount of lisp data (live + free lists) from
30M to 8M, and mallinfo().fordblks raises from 1M to 22M. This 21M of fragmented heap
is 'locked' into malloc freelists; they can be re-used, but will never released back to OS.

Dmitry

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Stefan Monnier said:

>> STIME   RSS    VSZ
>> Oct07 876524 1170572
>> Oct07 227016 499588
>
> I've just installed memory-usage.el as a package in the `elpa' branch
> (you'll soon be able to install it using package.el and in the mean time
> you can download it from http://bzr.savannah.gnu.org/lh/emacs/elpa/download/head:/memoryusage.el-20111021130523-o1v7kfuzrat3pcd9-2/memory-usage.el).
>
> It basically looks at the garbage-collect output and the various
> buffer's size to provide you with a human-readable description of the
> memory in use, from Elisp's point of view.  Don't expect too much of it,
> but it can be helpful.
>
> Could you try it out?

I am actually generating a gigantic amount of debugging output at GC
time using a patch from Dmitry Antipov right now: in a few days once
it's bloated up we'll see what we can see.

I'll try memory-usage.el out too once the bloating really kicks in:
I was looking for it earlier but then remembered that it's XEmacs that
has 'memory-usage-stats' as an interactive subr :)

> If this output does not explain the process's size you're seeing, then
> we have a leak in the C code somewhere.  If it does, then we have either
> a leak in Elisp, or at least an excessive memory use by some package,
> and hopefully we can at least figure out which category of object
> is involved.

... or the same thing XEmacs apparently had, which was excessive memory
consumption by one or more other still-unidentified C libraries.

>> Unfortunately that means, sigh, using our own malloc() again, which is
>> probably more painful than useful.
>
> Emacs already uses its own malloc for most objects: it uses malloc
> directly to allocate vectors (and this is actually something that we
> might want to change, because it (indirectly) comes with an O(log n)
> cost), but for strings, conses, markers, overlays, and floats it uses
> malloc only to get a chunk of memory which it then manages on its own.

True! So it wouldn't be as hellish to write our own allocator as it
would be for most projects, because we've done 90% of the work already
(including the hard part: allocating things of variable size).

> hash-tables, and a few more; and "markers" similarly include a few
> non-marker objects, mostly overlays.  Oh, and "intervals" are objects
> used to store text-properties.

Yeah. As an aside: most of the inside of Emacs is quite nice, but I am
seriously unimpressed by overlay storage. Linear scans for
'overlays-at'? Ew. One of these years I might try to implement something
better, like what XEmacs did for extents, so you can find extents over a
given buffer position more efficiently... (not because this is visibly a
problem in profiles that I've seen, merely because the inefficiency
offends me :) )

>> I suspect actually proving my contention first would be a good
>> idea.  Not sure how to get the addresses of Lisp objects from a running
>> Emacs though: gdb, presumably.
>
> I'd hack src/alloc.c to export the needed info to Elisp.  But maybe
> memory-usage will already give us enough info that this won't
> be necessary.

Already done by Dmitry before I could do more than blink :)

-- 
NULL && (void)

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Dmitry Antipov uttered the following:
> Nevertheless, glibc malloc design assumes that smaller (below
> threshold above) chunks may be still allocated with mmap().

Really? I double-checked the glibc-2.12 malloc source, and mmap()s only
used for large allocations above that minimum threshold, for multiple
(threaded) arenas, and if sbrk() fails. (It normally uses a sliding
threshold to define 'minimum', but that threshold only goes *up* from
the minimum, never down. However, because Emacs sets the threshold
explicitly, the dynamic behaviour is halted and mmap() should be used
for all requests >64Kb.)

btw, regarding

,----[ src/alloc.c ]
| /* Specify maximum number of areas to mmap.  It would be nice to use a
|    value that explicitly means "no limit".  */
| 
| #define MMAP_MAX_AREAS 100000000
`----

there is a reason glibc malloc uses a default value of 65536, which is
simply that older Linux kernels used to get rather unhappy with programs
that defined many tens of thousands of VMAs (mmapped regions, roughly).
I think the first two programs to trigger this were Java and
User-Mode Linux: the latter mmap()ed one 4Kb chunk per 4Kb of guest
memory, and soon ended up causing exponential slowdowns inside the
kernel memory allocator.

More recent kernels (anything in the 2.6 line, certainly) don't have
this problem, so I think the huge value Emacs uses is probably fine. (If
the problem recurs, you *will* hear about it: it's hard to miss it when
a text editor gets stuck inside the kernel for several seconds inside
a not-too-large mmap(). But I suspect anyone still running a kernel old
enough to have this problem also has too little memory to see it with
regions of the size Emacs will mmap().)

> Run emacs with:
>
> strace -e signal=none -e trace=mmap,mremap,munmap emacs
>
> and you will see a lot of allocation requests of very different sizes.

Yep. However, a lot of those may come from other libraries: a good few
of them mmap() data in from files (especially at startup), mmap() small
anonymous regions and the like.

> On the other side, lisp data is newer mapped, and in some cases this
> is the main reason which causes sbrk()'ed heap fragmentation. Don't
> beat toolkits, don't beat system libraries. Now I'm observing the case
> where 4 GCs reduces amount of lisp data (live + free lists) from 30M
> to 8M, and mallinfo().fordblks raises from 1M to 22M. This 21M of
> fragmented heap is 'locked' into malloc freelists; they can be
> re-used, but will never released back to OS.

That's exactly what I feared :(

-- 
NULL && (void)

Re: Emacs bzr memory footprint

[Dmitry Antipov]

On 10/21/2011 05:30 PM, Stefan Monnier wrote:

> Emacs already uses its own malloc for most objects: it uses malloc
> directly to allocate vectors (and this is actually something that we
> might want to change, because it (indirectly) comes with an O(log n)
> cost), but for strings, conses, markers, overlays, and floats it uses
> malloc only to get a chunk of memory which it then manages on its own.
> `memory-usage' does give you this kind of info when it says
> "5743392+864696 bytes in cons cells" which means 5MB of live cons-cells,
> and 800KB of cons-cells that are free (i.e. they are in a memory chunk
> that Emacs can't return to malloc because that chunk also contains live
> cons cells).

This creates two-level fragmentation: the whole heap is fragmented as seen
by malloc(), and allocated space is fragmented too, as seen by Emacs. Since
a lot of non-lisp allocation requests are satisfied from the same sbrk()ed
heap, here is the typical scenario: at startup, there is 10M of live lisp data.
Solid workload (e.g. visit 1000 files at once) increases this, say, to 100M.
When the workload is finished (kill all file-backed 1000 buffers), live lisp data
shrinks back to 10M. The rest 90M can't be freed because a) small percentage
is interleaved with live lisp data within the chunks managed by Emacs itself
and b) the rest, being freed by Emacs, can't be freed by malloc because
freed chunks are interleaved with non-freed, and all chunks are interleaved
with freed/lived lisp vectors and freed/lived non-lisp allocations. If next
solid workload will consume 100M of the heap, the most of this 90M will be re-used -
but it will never freed.

Due to all of the above, I suspect that the mostly-copying collector, where all
small (say, < 4K) lisp (and only lisp) allocation requests are satisfied from
1M-4M areas, might have substantial advantages over plain mark&sweep scheme.

Dmitry

Re: Emacs bzr memory footprint

[Stefan Monnier]

>> If this output does not explain the process's size you're seeing, then
>> we have a leak in the C code somewhere.  If it does, then we have either
>> a leak in Elisp, or at least an excessive memory use by some package,
>> and hopefully we can at least figure out which category of object
>> is involved.
> ... or the same thing XEmacs apparently had, which was excessive memory
> consumption by one or more other still-unidentified C libraries.

Yes, that's what I meant by "in the C code somewhere".  Another thread
recently seemed to suggest that maybe the new C-level support for gnutls
may cause such problems.

> True! So it wouldn't be as hellish to write our own allocator as it
> would be for most projects, because we've done 90% of the work already
> (including the hard part: allocating things of variable size).

Actually, no, that doesn't include the hard part: we only do our own
allocation for fixed-size objects (cons, floats, markers&overlays,
symbols, string headers) plus relocatable variable size objects (string
text).  For the non-relocatable, non-fixed size objects (i.e. vectors)
we defer to malloc.

> Yeah. As an aside: most of the inside of Emacs is quite nice, but I am
> seriously unimpressed by overlay storage. Linear scans for
> 'overlays-at'? Ew. One of these years I might try to implement something
> better, like what XEmacs did for extents, so you can find extents over a
> given buffer position more efficiently... (not because this is visibly a
> problem in profiles that I've seen, merely because the inefficiency
> offends me :) )

Yes, overlays (and to some extent markers) are in serious needs of
a redesign.  We get bitten by this performance problem, which is why
a lot of code goes through a fair bit of trouble to avoid overlays and
use text-properties instead (and the code that doesn't bother doing
that is usually only usable on small buffers).


        Stefan

Re: Emacs bzr memory footprint

[Stefan Monnier]

> This creates two-level fragmentation: the whole heap is fragmented as seen
> by malloc(), and allocated space is fragmented too, as seen by Emacs. Since

Yes.

> solid workload will consume 100M of the heap, the most of this 90M will be re-used -

Exactly: so this fragmentation is not a serious problem in the sense
that this free memory can be (re)used by Emacs.

> but it will never freed.

Right, so it's a problem in the sense that this memory will not be
available to other processes.

While this is not perfect, I don't consider it to be a serious problem.
E.g. when people complain about Emacs using a lot of memory it's always
(until now and I think in Nix's case as well) a case of too much memory
being used because of something like a leak, and not just "Emacs doesn't
return its free memory to the OS".

E.g. in the case of Nix, I think it's pretty clear that Emacs should
never have gotten to the point where it used up 800MB of memory.

> Due to all of the above, I suspect that the mostly-copying collector, where all
> small (say, < 4K) lisp (and only lisp) allocation requests are satisfied from
> 1M-4M areas, might have substantial advantages over plain mark&sweep scheme.

There's no argument that pretty anything else than mark&sweep is better
than mark&sweep in some significant respect.


        Stefan

Re: Emacs bzr memory footprint

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

[snip]

>> but it will never freed.
>
> Right, so it's a problem in the sense that this memory will not be
> available to other processes.
>
> While this is not perfect, I don't consider it to be a serious problem.
> E.g. when people complain about Emacs using a lot of memory it's always
> (until now and I think in Nix's case as well) a case of too much memory
> being used because of something like a leak, and not just "Emacs doesn't
> return its free memory to the OS".

The memory for the text of a buffer, is a single chunk? Or, more
generally, are large chunks of memory allocated for big buffers? If the
answer is yes, memory fragmentation may be problematic for those who
work with large buffers, unless the allocating algorithm avoids that.

[snip]

Re: Emacs bzr memory footprint

[Dmitry Antipov]

On 10/21/2011 07:22 PM, Nix wrote:

> Really? I double-checked the glibc-2.12 malloc source

Hmm... I didn't check the sources and didn't any tests, but info manual from 2.14 says:

`M_MMAP_THRESHOLD'
          All chunks larger than this value are allocated outside the
          normal heap, using the `mmap' system call.  This way it is
          guaranteed that the memory for these chunks can be returned
          to the system on `free'.  Note that requests smaller than
          this threshold might still be allocated via `mmap'.

That's why Emacs uses mallopt (M_MMAP_MAX, 0) each time before allocating
chunks for Lisp data - this effectively says 'no more mapped areas please',
regardless of the requested size.

The manual may be outdated, so I'll check this too. If you're right,
these calls of mallopt (M_MMAP_MAX, 0) becomes redundant with never glibcs
(but still should present for compatibility with old versions).

Dmitry

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Dmitry Antipov outgrape:

> On 10/21/2011 07:22 PM, Nix wrote:
>
>> Really? I double-checked the glibc-2.12 malloc source
>
> Hmm... I didn't check the sources and didn't any tests, but info manual from 2.14 says:
>
> `M_MMAP_THRESHOLD'
>          All chunks larger than this value are allocated outside the
>          normal heap, using the `mmap' system call.  This way it is
>          guaranteed that the memory for these chunks can be returned
>          to the system on `free'.  Note that requests smaller than
>          this threshold might still be allocated via `mmap'.

They might. If sbrk() fails, mmap() is tried instead before giving up.
However, in this case, resetting the mmap threshold isn't going to help:
it'll try anyway.

> The manual may be outdated, so I'll check this too. If you're right,
> these calls of mallopt (M_MMAP_MAX, 0) becomes redundant with never glibcs
> (but still should present for compatibility with old versions).

IIRC glibc malloc has always worked this way (in fact I think it worked
this way even back when it was Doug Lea's malloc, before Wolfram Gloger
took over maintenance). It's got less aggressive in use of mmap() over
the years, but has never allocated storage below the minimum threshold
except in dire straits, because mmap() is so slow. It certainly worked
this way when I first looked at it back in the 2.2 days.

(Though the calls are indeed harmless unless you want glibc's newer
auto-weighting, which I think we probably don't for Emacs as it would
make our existing fragmentation problems even worse.)

--- current bloat score, holding steady:

STIME     TIME   RSS    VSZ
13:38 00:12:07 374056 561116

-- GC165 end: lisp data 173241 Kb (144386 live, 28854 free), 139720 mem nodes (6549 Kb), heap 303948 Kb used, 78115 Kb free

(This sort of rising in bursts is also characteristic of heap
fragmentation.)

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 21 Oct 2011 20:27:28 +0200
> 
> The memory for the text of a buffer, is a single chunk?

Yes.

> Or, more generally, are large chunks of memory allocated for big
> buffers?

Yes.

> If the answer is yes, memory fragmentation may be problematic for
> those who work with large buffers, unless the allocating algorithm
> avoids that.

It does.

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Óscar Fuentes told this:
> The memory for the text of a buffer, is a single chunk? Or, more
> generally, are large chunks of memory allocated for big buffers? If the
> answer is yes, memory fragmentation may be problematic for those who
> work with large buffers, unless the allocating algorithm avoids that.

Emacs goes to some lengths to avoid allocating memory on the brk()ed
heap for buffers of significant size. The rules are somewhat arcane.

On a limited number of platforms (currently Cygwin, FreeBSD and IRIX
6.5), mmap() is used to map buffer text directly.

On platforms using the copy of Doug Lea's malloc() in src/gmalloc.c (or
using Doug Lea's malloc without being a GNU system, if any such systems
exist), buffers are allocated using the frankly terrifying
block-relocating memory allocator in ralloc.c, which uses sbrk()
directly and arranges for blocks allocated that way to dodge out of the
way of malloc() using the morecore_hook. (I'd be scared to trust my data
to this one.)

On GNU platforms which appear to be using Doug Lea's malloc(), we turn
this off and just use malloc() directly.

Emacs relies on all of these allocating the buffer storage in a single
(virtually) contiguous chunk (modulo the gap): the gap-motion code in
insdel.c, and much else, depends on being able to use simple pointer
arithmetic to move around the buffer.

-- 
NULL && (void)

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Stefan Monnier spake thusly:
>> True! So it wouldn't be as hellish to write our own allocator as it
>> would be for most projects, because we've done 90% of the work already
>> (including the hard part: allocating things of variable size).
>
> Actually, no, that doesn't include the hard part: we only do our own
> allocation for fixed-size objects (cons, floats, markers&overlays,
> symbols, string headers) plus relocatable variable size objects (string
> text).  For the non-relocatable, non-fixed size objects (i.e. vectors)
> we defer to malloc.

Oh heck. I assumed that were were allocating the strings themselves, not
just their headers, and didn't check the source code. Mea culpa.

>> Yeah. As an aside: most of the inside of Emacs is quite nice, but I am
>> seriously unimpressed by overlay storage. Linear scans for
>> 'overlays-at'? Ew. One of these years I might try to implement something
>> better, like what XEmacs did for extents, so you can find extents over a
>> given buffer position more efficiently... (not because this is visibly a
>> problem in profiles that I've seen, merely because the inefficiency
>> offends me :) )
>
> Yes, overlays (and to some extent markers) are in serious needs of
> a redesign.  We get bitten by this performance problem, which is why
> a lot of code goes through a fair bit of trouble to avoid overlays and
> use text-properties instead (and the code that doesn't bother doing
> that is usually only usable on small buffers).

Oh good. On my huge todo list it goes then!

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: Nix <nix@esperi.org.uk>
> Date: Fri, 21 Oct 2011 20:11:29 +0100
> Cc: emacs-devel@gnu.org
> 
> I'd be scared to trust my data to this one.

Don't be.  It's well tested on many platforms, and the Windows build
still uses it.  I don't think there are any bug reports related to it.

> Emacs relies on all of these allocating the buffer storage in a single
> (virtually) contiguous chunk (modulo the gap)

The buffer text is a contiguous chunk of memory; the gap is just a
portion of that chunk ready to receive inserted text.

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Eli Zaretskii said:

>> From: Nix <nix@esperi.org.uk>
>> Date: Fri, 21 Oct 2011 20:11:29 +0100
>> Cc: emacs-devel@gnu.org
>> 
>> I'd be scared to trust my data to this one.
>
> Don't be.  It's well tested on many platforms, and the Windows build
> still uses it.  I don't think there are any bug reports related to it.

OK. Doesn't make it less terrifying though. I don't think I've ever seen
a clearer example of duelling allocators. :)

>> Emacs relies on all of these allocating the buffer storage in a single
>> (virtually) contiguous chunk (modulo the gap)
>
> The buffer text is a contiguous chunk of memory; the gap is just a
> portion of that chunk ready to receive inserted text.

Exactly. (I think I've implemented buffer-gap-based algorithms three
separate times: they seems kludgy, but are so much simpler than pretty
much every alternative, and are fast in all but the most uncommon
cases.)

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Fri, Oct 21, 2011 at 8:00 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> > This creates two-level fragmentation: the whole heap is fragmented as seen
> > by malloc(), and allocated space is fragmented too, as seen by Emacs. Since
>
> Yes.
>
> > solid workload will consume 100M of the heap, the most of this 90M will be re-used -
>
> Exactly: so this fragmentation is not a serious problem in the sense
> that this free memory can be (re)used by Emacs.
>
> > but it will never freed.
>
> Right, so it's a problem in the sense that this memory will not be
> available to other processes.
>
> While this is not perfect, I don't consider it to be a serious problem.
> E.g. when people complain about Emacs using a lot of memory it's always
> (until now and I think in Nix's case as well) a case of too much memory
> being used because of something like a leak, and not just "Emacs doesn't
> return its free memory to the OS".

It is a problem for everybody like me who doesn't have
gigs of free RAM. Sorry.

I'll remember to --disable-gnutls and not use Gnus for my
own safety, for the moment. That's a good catch and seems
to help.

Re: Emacs bzr memory footprint

[Stephen J. Turnbull]

Carsten Mattner writes:
 > On Fri, Oct 21, 2011 at 8:00 PM, Stefan Monnier
 > <monnier@iro.umontreal.ca> wrote:

 > > While this is not perfect, I don't consider it to be a serious problem.
 > > E.g. when people complain about Emacs using a lot of memory it's always
 > > (until now and I think in Nix's case as well) a case of too much memory
 > > being used because of something like a leak, and not just "Emacs doesn't
 > > return its free memory to the OS".
 > 
 > It is a problem for everybody like me who doesn't have
 > gigs of free RAM. Sorry.

Nobody is saying that running out of memory isn't a serious problem.
Stefan's point is simply that "if you don't have enough RAM to use
Emacs comfortably, you don't have enough RAM to use Emacs comfortably."

There are applications that use huge amounts of memory at
initialization or in the early part of an algorithm, then can free it
and never need it again.  In those applications, freeing memory after
use helps a lot.  Emacs isn't one of those; average memory usage is
normally a reasonably high fraction of peak usage (at least 75%, often
effectively 100%, in my own usage patterns) and the peaks tend to
recur.  If that's more memory than you can afford, there's little we
can do.  Probably memory usage can be decreased on average or even at
peak by a few percent, but is it worth the effort to develop and
maintain very sophisticated allocators?  While it is my belief that
XEmacs's current excess memory consumption is related to system
libraries like X, I have to admit that reports started to appear about
the same time that a new garbage collection framework was introduced
-- it's quite possible that the new GC is buggy!

On the other hand, memory leaks of various kinds are bugs that should
be fixed when understood, and in the meantime features that trigger
them identified so you can avoid them.

Re: Emacs bzr memory footprint

[Sven Joachim]

On 2011-10-22 09:34 +0200, Carsten Mattner wrote:

> On Fri, Oct 21, 2011 at 8:00 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>> > This creates two-level fragmentation: the whole heap is fragmented as seen
>> > by malloc(), and allocated space is fragmented too, as seen by Emacs. Since
>>
>> Yes.
>>
>> > solid workload will consume 100M of the heap, the most of this 90M will be re-used -
>>
>> Exactly: so this fragmentation is not a serious problem in the sense
>> that this free memory can be (re)used by Emacs.
>>
>> > but it will never freed.
>>
>> Right, so it's a problem in the sense that this memory will not be
>> available to other processes.
>>
>> While this is not perfect, I don't consider it to be a serious problem.
>> E.g. when people complain about Emacs using a lot of memory it's always
>> (until now and I think in Nix's case as well) a case of too much memory
>> being used because of something like a leak, and not just "Emacs doesn't
>> return its free memory to the OS".
>
> It is a problem for everybody like me who doesn't have
> gigs of free RAM. Sorry.
>
> I'll remember to --disable-gnutls and not use Gnus for my
> own safety, for the moment. That's a good catch and seems
> to help.

Actually, Gnus is not a memory hog as long as you build Emacs
"--without-gnutls".  But with gnutls support, it's clearly leaking
memory.

Cheers,
       Sven

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Sat, Oct 22, 2011 at 10:44 AM, Sven Joachim <svenjoac@gmx.de> wrote:
> Actually, Gnus is not a memory hog as long as you build Emacs
> "--without-gnutls".  But with gnutls support, it's clearly leaking
> memory.

Ah, I see. Thanks for the clarification, Sven.

Both gnus and erc-tls seem to also work with a forked openssl or gnutls
process instead of the linked-in gnutls lib, if I needed.
So I think I will avoid linking gnutls for the moment.
I have switched to irssi for IRC anyway.

Re: Emacs bzr memory footprint

[Carsten Mattner]

On Sat, Oct 22, 2011 at 10:30 AM, Stephen J. Turnbull
<stephen@xemacs.org> wrote:
> Nobody is saying that running out of memory isn't a serious problem.
> Stefan's point is simply that "if you don't have enough RAM to use
> Emacs comfortably, you don't have enough RAM to use Emacs comfortably."

Sure.

Actually, if I could port the keybindings which work well with
X or Cocoa Emacs clients to the terminal emulator (-nw), I could
save even more memory.

Any tutorial/gotchas I can study to make my keybindings portable
between xterm, tmux, screen, Cocoa and X (no toolkit)?

The color discrepancy is a minor issue, which can be resolved
with selecting a theme that also looks correct in the term emulator.

> There are applications that use huge amounts of memory at
> initialization or in the early part of an algorithm, then can free it
> and never need it again.  In those applications, freeing memory after
> use helps a lot.  Emacs isn't one of those; average memory usage is
> normally a reasonably high fraction of peak usage (at least 75%, often
> effectively 100%, in my own usage patterns) and the peaks tend to
> recur.  If that's more memory than you can afford, there's little we
> can do.  Probably memory usage can be decreased on average or even at
> peak by a few percent, but is it worth the effort to develop and
> maintain very sophisticated allocators?  While it is my belief that
> XEmacs's current excess memory consumption is related to system
> libraries like X, I have to admit that reports started to appear about
> the same time that a new garbage collection framework was introduced
> -- it's quite possible that the new GC is buggy!

A GC that doesn't fit the usage patterns may be worse than
no GC at all :). There's a reason we have many GC algos.
Erlang's copying GC fits the many-lightweight processes model
wel, for example. It's hard to use such a GC with the JVM's usual
workloads or GHC's, although GHC has and Erlang process like
the surrounding mechanics and different models included in GHC
make it harder. This is a case where elisp and Erlang have it easier,
I believe.

> On the other hand, memory leaks of various kinds are bugs that should
> be fixed when understood, and in the meantime features that trigger
> them identified so you can avoid them.

Is the gnutls memory usage a leak, or just some allocated arrays for
crypto arrays? Just a wild guess, nothing scientific to back it up.

Re: Emacs bzr memory footprint

[Stephen J. Turnbull]

Carsten Mattner writes:

 > Is the gnutls memory usage a leak, or just some allocated arrays for
 > crypto arrays? Just a wild guess, nothing scientific to back it up.

It sounds like a leak to me, since the reports we're seeing are of
memory creeping up over time.  I would expect any arrays to be on the
order of a few MB, not 100s.

Re: Emacs bzr memory footprint

[Stefan Monnier]

> It is a problem for everybody like me who doesn't have
> gigs of free RAM. Sorry.

No, it's not.  You misunderstand.  The problem that I say is not
important is the fact that if ever Emacs gets up to N megabytes for good
reasons (you really use up that much stuff at a given time), and later
on you use up less stuff, the Emacs process may keep using up
N megabytes (most of it inactive).

If you have little memory and N megabytes is a problem, then it will be
a problem already when you do use that much, i.e. before the "waste"
comes into play.

FWIW, my office desktop is a 1GB fit-pc2.

> I'll remember to --disable-gnutls and not use Gnus for my own safety,
> for the moment.  That's a good catch and seems to help.

Looks like we have a real leak there, yes.


        Stefan

Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Stefan Monnier uttered the following:

> I've just installed memory-usage.el as a package in the `elpa' branch
> (you'll soon be able to install it using package.el and in the mean time
> you can download it from http://bzr.savannah.gnu.org/lh/emacs/elpa/download/head:/memoryusage.el-20111021130523-o1v7kfuzrat3pcd9-2/memory-usage.el).
>
> It basically looks at the garbage-collect output and the various
> buffer's size to provide you with a human-readable description of the
> memory in use, from Elisp's point of view.  Don't expect too much of it,
> but it can be helpful.
>
> Could you try it out?

Two rounds of stats. Firstly, absolute peak legitimate memory usage,
when reading a truly massive nonexpired newsgroup:

Garbage collection stats:
((1682002 . 1142162) (78246 . 155) (25344 . 17561) 88139150 2009895 (2439 . 9353) (32882 . 12331) (452963 . 384130))

 =>	26912032+18274592 bytes in cons cells
	3755808+7440 bytes in symbols
	1013760+702440 bytes in markers
	39024+149648 bytes in floats
	1841392+690536 bytes in intervals
	14494816+12292160 bytes in string headers
	88139150 bytes of string chars
	88139150 bytes of vector slots

Total bytes in lisp objects: 170322693 (live 138205877, dead 32116816)

Buffer ralloc memory usage:
108 buffers
7086099 bytes total (60682 in gaps)
[...]


In normal usage, with Huge Newsgroup closed but with my usual pair of
nnml and gnutls-backed nnimap groups open (after another GC to push
memory usage down properly):

Garbage collection stats:
((1678832 . 1131753) (77833 . 463) (24737 . 17683) 88021904 2005723 (2428 . 9364) (32180 . 14186) (449983 . 387110))

 =>	26861312+18108048 bytes in cons cells
	3735984+22224 bytes in symbols
	989480+707320 bytes in markers
	38848+149824 bytes in floats
	1802080+794416 bytes in intervals
	14399456+12387520 bytes in string headers
	88021904 bytes of string chars
	88021904 bytes of vector slots

Total bytes in lisp objects: 170024139 (live 137854787, dead 32169352)

Buffer ralloc memory usage:
104 buffers
6977389 bytes total (55669 in gaps)
      Size	Gap	Name

STIME     TIME   RSS    VSZ
Oct21 00:55:23 524568 716588

So I have 170Mb in Lisp objects, 7Mb in buffers... and a 716Mb VSZ.
500Mb of that is heap, too:

% pmap 2676 | grep anon
0000000002fc2000 499180K rw---    [ anon ]
00007fcd0017b000      4K rw---    [ anon ]
00007fcd05fbf000      4K rw---    [ anon ]
00007fcd08d72000    304K rw---    [ anon ]
00007fcd08e68000    348K rw---    [ anon ]
00007fcd094d1000   1300K rw---    [ anon ]
00007fcd097c6000      4K rw---    [ anon ]
00007fcd0ab36000     96K rw---    [ anon ]
00007fcd0ab9c000    436K rw---    [ anon ]
00007fcd0ac12000    424K rw---    [ anon ]
00007fcd0ac81000    192K rw---    [ anon ]
00007fcd0adea000    152K rw---    [ anon ]
00007fcd0ae23000    256K rw---    [ anon ]
00007fcd0aef9000     84K rw---    [ anon ]
00007fcd0dd80000      4K rw---    [ anon ]
00007fcd0e975000      4K rw---    [ anon ]
00007fcd0f1d7000      8K rw---    [ anon ]
00007fcd0f8db000     12K rw---    [ anon ]
00007fcd10ef8000     20K rw---    [ anon ]
00007fcd12041000      4K rw---    [ anon ]
00007fcd1225a000     16K rw---    [ anon ]
00007fcd12919000    100K rw---    [ anon ]
00007fcd12f1d000      4K rw---    [ anon ]
00007fcd1315e000      4K rw---    [ anon ]
00007fcd13672000      8K rw---    [ anon ]
00007fcd14df3000     64K rw---    [ anon ]
00007fcd15280000     16K rw---    [ anon ]
00007fcd156f1000      4K rw---    [ anon ]
00007fcd15b68000      8K rw---    [ anon ]
00007fcd15d38000    112K rw---    [ anon ]
00007fcd15d85000      8K rw---    [ anon ]
00007fcd15d89000      4K rw---    [ anon ]
00007fffca9ff000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]

Looky looky, leak.

> If this output does not explain the process's size you're seeing, then
> we have a leak in the C code somewhere.

Oh yes.

> I'd hack src/alloc.c to export the needed info to Elisp.  But maybe
> memory-usage will already give us enough info that this won't
> be necessary.

Here's the last summary line from Dmitry Antipov's patch written for
exactly this purpose, which emits a *lot* of other usage info too
(if you want a 5Mb upload I can give you the whole log, lzmaed, but
look out, uncompressed it comes to 264M):

-- GC507 end: lisp data 179658 Kb (148315 live, 31342 free), 152325 mem
   nodes (7140 Kb), heap 418617 Kb used, 117794 Kb free

That comes from this printf():

+  fprintf (stderr, 
+	   "-- GC%d end: lisp data %d Kb (%d live, %d free), "
+#if GC_MARK_STACK || defined GC_MALLOC_CHECK
+	   "%d mem nodes (%d Kb), "
+#endif
+	   "heap %d Kb used, %d Kb free\n",
+	   /* GC count.  */
+	   gcs_done,
+	   /* Total Lisp data.  */
+	   ((total_conses + total_free_conses) * sizeof (struct Lisp_Cons) +
+	    (total_markers + total_free_markers) * sizeof (union Lisp_Misc) +
+	    (total_symbols + total_free_symbols) * sizeof (struct Lisp_Symbol) +
+	    total_string_size +
+	    total_vector_size * sizeof (Lisp_Object) +
+	    (total_floats + total_free_floats) * sizeof (struct Lisp_Float) +
+	    (total_intervals + total_free_intervals) * sizeof (struct interval) +
+	    (total_strings + total_free_strings) * sizeof (struct Lisp_String)) / 1024,
+	   /* Live Lisp data.  */
+	   (total_conses * sizeof (struct Lisp_Cons) +
+	    total_markers * sizeof (union Lisp_Misc) +
+	    total_symbols * sizeof (struct Lisp_Symbol) +
+	    total_string_size +
+	    total_vector_size * sizeof (Lisp_Object) +
+	    total_floats * sizeof (struct Lisp_Float) +
+	    total_intervals * sizeof (struct interval) +
+	    total_strings * sizeof (struct Lisp_String)) / 1024,
+	   /* Free Lisp data.  */
+	   (total_free_conses * sizeof (struct Lisp_Cons) +
+	    total_free_markers * sizeof (union Lisp_Misc) +
+	    total_free_symbols * sizeof (struct Lisp_Symbol) +
+	    total_free_floats * sizeof (struct Lisp_Float) +
+	    total_free_intervals * sizeof (struct interval) +
+	    total_free_strings * sizeof (struct Lisp_String)) / 1024,
+#if GC_MARK_STACK || defined GC_MALLOC_CHECK
+	   total_mem_nodes, total_mem_nodes * sizeof (struct mem_node) / 1024,
+#endif
+#ifdef DOUG_LEA_MALLOC
+	   mallinfo ().uordblks / 1024, mallinfo ().fordblks / 1024);
+#else
+	   0, 0);
+#endif

Re: Emacs bzr memory footprint

[Stefan Monnier]

> So I have 170Mb in Lisp objects, 7Mb in buffers... and a 716Mb VSZ.
> 500Mb of that is heap, too:

>> If this output does not explain the process's size you're seeing, then
>> we have a leak in the C code somewhere.
> Oh yes.

Do you use the libgnutls bindings, by any chance?
If so, can you try to avoid using it and just stick to
gnutls-cli instead?


        Stefan

Re: Emacs bzr memory footprint

[Nix]

On 25 Oct 2011, Stefan Monnier stated:

>>> If this output does not explain the process's size you're seeing, then
>>> we have a leak in the C code somewhere.
>> Oh yes.
>
> Do you use the libgnutls bindings, by any chance?

Yes.

> If so, can you try to avoid using it and just stick to
> gnutls-cli instead?

It looks like doing that is a simple matter of compiling it out and
open-network-stream will fall back automatically, so, sure, I'll do
that next time I rebuild (probably tomorrow), let it run for a few
days, and see if it bloats.

-- 
NULL && (void)

gnutls memory leak [Was: Re: Emacs bzr memory footprint]

[Chong Yidong]

I think I see a problem in gnutls.c.  The functions for freeing the
credentials structures,

 fn_gnutls_certificate_free_credentials (x509_cred);
 fn_gnutls_anon_free_client_credentials (anon_cred);

are only called at gnutls.c:761 and gnutls.c:767, specifically for the
case where Fgnutls_boot is called on a process that already has
credentials structures (gnutls_x509_cred or gnutls_anon_cred) allocated.
IIUC, those freeing functions don't get called when the process is
killed via kill-process.

Could someone who had a hand in gnutls.c check if this is accurate?  If
not, I can try to put together a patch.

BTW, does anyone know a simple test-case for using the gnutls support,
to make it easier to test for the leak?

Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]

[Chong Yidong]

Since no one responded so far, I went ahead and committed a fix for that
particular memory leak.  I verified that with this fix, the credentials
structures are deallocated after sending a mail via SSL; previously they
won't.

If someone could check whether the memory leak is gone or still
persists, that would be great.

Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]

[Ted Zlatanov]

On Thu, 27 Oct 2011 14:10:38 +0800 Chong Yidong <cyd@gnu.org> wrote: 

CY> Since no one responded so far, I went ahead and committed a fix for that
CY> particular memory leak.  I verified that with this fix, the credentials
CY> structures are deallocated after sending a mail via SSL; previously they
CY> won't.

CY> If someone could check whether the memory leak is gone or still
CY> persists, that would be great.

Thanks, I wasn't able to respond in time.  But I doubt the credentials
are the problem.  They don't get allocated/deallocated all that often,
unless I've misunderstood something fundamental in the GnuTLS code.  I
suspect the error logging and general packet handling code.

Ted

Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]

[Ted Zlatanov]

On Wed, 26 Oct 2011 22:52:21 +0800 Chong Yidong <cyd@gnu.org> wrote: 

CY> BTW, does anyone know a simple test-case for using the gnutls support,
CY> to make it easier to test for the leak?

gnutls.el has some in the comments:

;; (open-gnutls-stream "tls" "tls-buffer" "yourserver.com" "https")
;; (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")

Then you can just kill that buffer.

Ted

Re: Emacs bzr memory footprint

[Ted Zlatanov]

On Sun, 23 Oct 2011 01:08:15 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Carsten Mattner writes:
>> Is the gnutls memory usage a leak, or just some allocated arrays for
>> crypto arrays? Just a wild guess, nothing scientific to back it up.

SJT> It sounds like a leak to me, since the reports we're seeing are of
SJT> memory creeping up over time.  I would expect any arrays to be on the
SJT> order of a few MB, not 100s.

We have 3 reports from Sven Joachim, Nix, and Carsten Mattner.  I will
do my best to find the issues.  They seem to be in the C code I wrote to
link with GnuTLS.

It would be really helpful if someone who knows C debugging well could
run Valgrind or something similar against Emacs with GnuTLS compiled,
hitting all these leaks.  I will take a look at the code, since I am
also reviewing it to prevent the lockups on hung connections, but
concrete Valgrind reports would be great.

Thanks
Ted

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 27 Oct 2011 18:10:56 -0400
> 
> It would be really helpful if someone who knows C debugging well could
> run Valgrind or something similar against Emacs with GnuTLS compiled,
> hitting all these leaks.  I will take a look at the code, since I am
> also reviewing it to prevent the lockups on hung connections, but
> concrete Valgrind reports would be great.

Alas, I think Emacs will not run under valgrind.

Re: Emacs bzr memory footprint

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Alas, I think Emacs will not run under valgrind.

XEmacs does (at least some versions have done so ;-) so it's not
impossible in principle AFAIK.  Maybe valgrind technology has improved
since you last tried, or maybe it only works on particular
architectures, but I'm pretty sure a couple of people were able to
valgrind XEmacs.

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 28 Oct 2011 18:34:23 +0900
> Cc: emacs-devel@gnu.org
> 
> Eli Zaretskii writes:
> 
>  > Alas, I think Emacs will not run under valgrind.
> 
> XEmacs does (at least some versions have done so ;-) so it's not
> impossible in principle AFAIK.

Nothing is impossible in principle: this is software.  It just could
be impractical or extremely hard.

One of the problems is dumping, and that of XEmacs uses an entirely
different method AFAIK.  Maybe people should try running temacs
instead.

> Maybe valgrind technology has improved since you last tried, or
> maybe it only works on particular architectures, but I'm pretty sure
> a couple of people were able to valgrind XEmacs.

It definitely is worth trying again every now and then, yes.

If someone succeeds, please add the necessary know-how to etc/DEBUG.

Re: Emacs bzr memory footprint

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > One of the problems is dumping, and that of XEmacs uses an entirely
 > different method AFAIK.

True.  I was just saying....

 > Maybe people should try running temacs instead.

Ah, that rings a bell.  Indeed, running from temacs may have been
necessary.  I don't know about Emacs, but XEmacs has a recipe for
running a regular session from temacs in the Makefile (it's something
like "make run-temacs").  Emacs is probably different, but maybe you
already have the necessary magic spells for doing this?  Or maybe it's
trivial for Emacs (it requires some special options for XEmacs, no big
deal but they're less than obvious or well-documented ;).

IIRC this takes a lot of extra time at startup, but once the dumpable
Lisp is loaded, temacs is otherwise no slower (but valgrind does cost
some performance).

Re: Emacs bzr memory footprint

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 19:22:14 +0900
> 
>  > Maybe people should try running temacs instead.
> 
> Ah, that rings a bell.  Indeed, running from temacs may have been
> necessary.  I don't know about Emacs, but XEmacs has a recipe for
> running a regular session from temacs in the Makefile (it's something
> like "make run-temacs").  Emacs is probably different, but maybe you
> already have the necessary magic spells for doing this?  Or maybe it's
> trivial for Emacs (it requires some special options for XEmacs, no big
> deal but they're less than obvious or well-documented ;).

Dan made temacs runnable as you'd run any program.  So I imagine
one needs to do this (100% untested):

  ./temacs -l loadup
  M-x load-file RET ~/.emacs RET

(It is possible that the second line is not needed, but I guess anyone
should be able to see immediately if her customizations are or aren't
in effect.)

And thereafter use Emacs as usual.

Modifications of the above recipe to run under valgrind are left to
the interested parties ;-)

valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

> So I imagine one needs to do this (100% untested):
>
>   ./temacs -l loadup
>   M-x load-file RET ~/.emacs RET

Actually
 ./temacs 

is enough.

On Fedora 15 with valgrind-3.6.1 it is possible to run 

valgrind ./temacs

Valgrind produces these warnings, maybe some of them are interesting:

==30382== Conditional jump or move depends on uninitialised value(s)
==30382==    at 0x5B7084: mark_object (alloc.c:5691)
==30382==    by 0x5B6E71: mark_object (alloc.c:5612)
==30382==    by 0x5B6DDF: mark_object (alloc.c:5591)
==30382==    by 0x5B7109: mark_object (alloc.c:5702)
==30382==    by 0x5B6E1D: mark_object (alloc.c:5595)
==30382==    by 0x5B687B: mark_vectorlike (alloc.c:5391)
==30382==    by 0x5B6DA8: mark_object (alloc.c:5579)
==30382==    by 0x5B6DDF: mark_object (alloc.c:5591)
==30382==    by 0x5B687B: mark_vectorlike (alloc.c:5391)
==30382==    by 0x5B6DA8: mark_object (alloc.c:5579)
==30382==    by 0x5B6DDF: mark_object (alloc.c:5591)
==30382==    by 0x5B687B: mark_vectorlike (alloc.c:5391)
==30382== 

==30382== Conditional jump or move depends on uninitialised value(s)
==30382==    at 0x4731E6: note_mouse_highlight (xdisp.c:26841)
==30382==    by 0x4F8682: note_mouse_movement (xterm.c:3830)
==30382==    by 0x4FCB2E: handle_one_xevent (xterm.c:6652)
==30382==    by 0x4FD792: XTread_socket (xterm.c:7133)
==30382==    by 0x5437F7: read_avail_input (keyboard.c:6821)
==30382==    by 0x54409F: handle_async_input (keyboard.c:7149)
==30382==    by 0x5440BE: process_pending_signals (keyboard.c:7165)
==30382==    by 0x62D713: wait_reading_process_output (process.c:4340)
==30382==    by 0x419ED2: sit_for (dispnew.c:5971)
==30382==    by 0x53AF87: read_char (keyboard.c:2687)
==30382==    by 0x548876: read_key_sequence (keyboard.c:9290)
==30382==    by 0x538709: command_loop_1 (keyboard.c:1447)

==30382== Conditional jump or move depends on uninitialised value(s)
==30382==    at 0x46D785: x_write_glyphs (xdisp.c:24661)
==30382==    by 0x413E4A: update_text_area (dispnew.c:3765)
==30382==    by 0x414794: update_window_line (dispnew.c:4006)
==30382==    by 0x413A1A: update_window (dispnew.c:3625)
==30382==    by 0x413256: update_single_window (dispnew.c:3370)
==30382==    by 0x511A93: Fx_show_tip (xfns.c:5189)
==30382==    by 0x5D6E47: Ffuncall (eval.c:2995)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x6221C6: Fbyte_code (bytecode.c:423)
==30382==    by 0x5D56AB: eval_sub (eval.c:2328)
==30382==    by 0x5D381A: internal_lisp_condition_case (eval.c:1453)
==30382==    by 0x62392A: exec_byte_code (bytecode.c:981)


==30382== Conditional jump or move depends on uninitialised value(s)
==30382==    at 0x5B489D: live_misc_p (alloc.c:4001)
==30382==    by 0x5B4C5D: mark_maybe_object (alloc.c:4138)
==30382==    by 0x5B4FE6: mark_memory (alloc.c:4274)
==30382==    by 0x5B507E: mark_stack (alloc.c:4532)
==30382==    by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
==30382==    by 0x5D69F0: Ffuncall (eval.c:2911)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)


==30382== Conditional jump or move depends on uninitialised value(s)
==30382==    at 0x499251: coding_set_destination (coding.c:1046)
==30382==    by 0x4B477A: decode_coding (coding.c:7022)
==30382==    by 0x4B7317: decode_coding_object (coding.c:7671)
==30382==    by 0x63610F: Fcall_process (callproc.c:813)
==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
==30382==    by 0x5D5ECA: Fapply (eval.c:2479)
==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)

==30382== Conditional jump or move depends on uninitialised value(s)
==30382==    at 0x6302A9: send_process (process.c:5398)
==30382==    by 0x630DC8: Fprocess_send_string (process.c:5648)
==30382==    by 0x5D6D61: Ffuncall (eval.c:2977)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)

==30382== Conditional jump or move depends on uninitialised value(s)
==30382==    at 0x636022: Fcall_process (callproc.c:799)
==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
==30382==    by 0x5D5ABB: Fapply (eval.c:2422)
==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382==    by 0x5D5ABB: Fapply (eval.c:2422)
==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)

==30382== Use of uninitialised value of size 8
==30382==    at 0x5B7000: mark_object (alloc.c:5674)
==30382==    by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
==30382==    by 0x5B4FE6: mark_memory (alloc.c:4274)
==30382==    by 0x5B507E: mark_stack (alloc.c:4532)
==30382==    by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
==30382==    by 0x5D69F0: Ffuncall (eval.c:2911)
==30382==    by 0x5D64CA: call2 (eval.c:2758)
==30382==    by 0x5809AA: Finsert_file_contents (fileio.c:4144)
==30382==    by 0x5D6E00: Ffuncall (eval.c:2990)
==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)

Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]

[Ted Zlatanov]

On Thu, 27 Oct 2011 18:01:47 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Thu, 27 Oct 2011 14:10:38 +0800 Chong Yidong <cyd@gnu.org> wrote: 
CY> Since no one responded so far, I went ahead and committed a fix for that
CY> particular memory leak.  I verified that with this fix, the credentials
CY> structures are deallocated after sending a mail via SSL; previously they
CY> won't.

CY> If someone could check whether the memory leak is gone or still
CY> persists, that would be great.

TZ> Thanks, I wasn't able to respond in time.  But I doubt the credentials
TZ> are the problem.  They don't get allocated/deallocated all that often,
TZ> unless I've misunderstood something fundamental in the GnuTLS code.  I
TZ> suspect the error logging and general packet handling code.

After comparing an Emacs with and without your fix, actually, I see a
2MB difference in the memory usage every time a GnuTLS connection is
opened.  That may have been the culprit, indeed.

I'm running the new Emacs and trying to balloon the memory usage but so
far have been unsuccessful.

Nix, Carsten, Sven, could you give a recent build a try?

Ted

valgrind warnings [Re: Emacs bzr memory footprint]

[Stephen J. Turnbull]

Dan Nicolaescu writes:

 > Actually
 >  ./temacs 
 > 
 > is enough.

Nice work!

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

Dan Nicolaescu <dann@gnu.org> writes:


> valgrind ./temacs
>
> Valgrind produces these warnings, maybe some of them are interesting:
>
> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382==    at 0x4731E6: note_mouse_highlight (xdisp.c:26841)
> ==30382==    by 0x4F8682: note_mouse_movement (xterm.c:3830)
> ==30382==    by 0x4FCB2E: handle_one_xevent (xterm.c:6652)
> ==30382==    by 0x4FD792: XTread_socket (xterm.c:7133)
> ==30382==    by 0x5437F7: read_avail_input (keyboard.c:6821)
> ==30382==    by 0x54409F: handle_async_input (keyboard.c:7149)
> ==30382==    by 0x5440BE: process_pending_signals (keyboard.c:7165)
> ==30382==    by 0x62D713: wait_reading_process_output (process.c:4340)
> ==30382==    by 0x419ED2: sit_for (dispnew.c:5971)
> ==30382==    by 0x53AF87: read_char (keyboard.c:2687)
> ==30382==    by 0x548876: read_key_sequence (keyboard.c:9290)
> ==30382==    by 0x538709: command_loop_1 (keyboard.c:1447)

This is because window_from_coordinates in note_mouse_highlight is
passed "&part", where "part" is an uninitialized local variable.
When !NILP(window) "part" is not touched, so it stays uninitialized.
Can someone who understands this code fix this please?

> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382==    at 0x46D785: x_write_glyphs (xdisp.c:24661)
> ==30382==    by 0x413E4A: update_text_area (dispnew.c:3765)
> ==30382==    by 0x414794: update_window_line (dispnew.c:4006)
> ==30382==    by 0x413A1A: update_window (dispnew.c:3625)
> ==30382==    by 0x413256: update_single_window (dispnew.c:3370)
> ==30382==    by 0x511A93: Fx_show_tip (xfns.c:5189)
> ==30382==    by 0x5D6E47: Ffuncall (eval.c:2995)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x6221C6: Fbyte_code (bytecode.c:423)
> ==30382==    by 0x5D56AB: eval_sub (eval.c:2328)
> ==30382==    by 0x5D381A: internal_lisp_condition_case (eval.c:1453)
> ==30382==    by 0x62392A: exec_byte_code (bytecode.c:981)

I think I fixed this one.

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Ted Zlatanov]

On Fri, 28 Oct 2011 07:52:12 -0400 Dan Nicolaescu <dann@gnu.org> wrote: 
DN> On Fedora 15 with valgrind-3.6.1 it is possible to run 

DN> valgrind ./temacs

I ran this and opened and closed a GnuTLS connection many times:

(open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")

I did not see any errors printed when I did this, but on exit a lot of
information was printed that did not seem related (I could not analyze
it all, it was too much data and I don't know what to look for other
than "gnutls").  Is this enough to tell us there are no memory leaks in
the "open connection" part of the code?  I'll test sending and receiving
data if so.

Ted

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Chong Yidong]

Dan Nicolaescu <dann@gnu.org> writes:

> This is because window_from_coordinates in note_mouse_highlight is
> passed "&part", where "part" is an uninitialized local variable.
> When !NILP(window) "part" is not touched, so it stays uninitialized.
> Can someone who understands this code fix this please?

Fixed; thanks.

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@gnu.org>
> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>,  emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 10:38:03 -0400
> 
> > Valgrind produces these warnings, maybe some of them are interesting:
> >
> > ==30382== Conditional jump or move depends on uninitialised value(s)
> > ==30382==    at 0x4731E6: note_mouse_highlight (xdisp.c:26841)
> > ==30382==    by 0x4F8682: note_mouse_movement (xterm.c:3830)
> > ==30382==    by 0x4FCB2E: handle_one_xevent (xterm.c:6652)
> > ==30382==    by 0x4FD792: XTread_socket (xterm.c:7133)
> > ==30382==    by 0x5437F7: read_avail_input (keyboard.c:6821)
> > ==30382==    by 0x54409F: handle_async_input (keyboard.c:7149)
> > ==30382==    by 0x5440BE: process_pending_signals (keyboard.c:7165)
> > ==30382==    by 0x62D713: wait_reading_process_output (process.c:4340)
> > ==30382==    by 0x419ED2: sit_for (dispnew.c:5971)
> > ==30382==    by 0x53AF87: read_char (keyboard.c:2687)
> > ==30382==    by 0x548876: read_key_sequence (keyboard.c:9290)
> > ==30382==    by 0x538709: command_loop_1 (keyboard.c:1447)
> 
> This is because window_from_coordinates in note_mouse_highlight is
> passed "&part", where "part" is an uninitialized local variable.
> When !NILP(window) "part" is not touched, so it stays uninitialized.

That's not my reading of the code.  Here's what happens in
window_from_coordinates:

  window = Qnil;
  cw.window = &window, cw.x = x, cw.y = y; cw.part = part;
  foreach_window (f, check_window_containing, &cw);

So cw.part now holds "part", which is a pointer.
check_window_containing does this:

  found = coordinates_in_window (w, cw->x, cw->y);
  if (found != ON_NOTHING)
    {
      *cw->part = found;
      XSETWINDOW (*cw->window, w);

So if it finds a window, it touches "part" through cw->part and sets
`window' (through cw->window) to a non-nil value.  Now back in
window_from_coordinates:

  if (NILP (window)
      && tool_bar_p
      && WINDOWP (f->tool_bar_window)
      && WINDOW_TOTAL_LINES (XWINDOW (f->tool_bar_window)) > 0
      && (coordinates_in_window (XWINDOW (f->tool_bar_window), x, y)
	  != ON_NOTHING))
    {
      *part = ON_TEXT;
      window = f->tool_bar_window;
    }

If `window' is non-nil, "part" was already set inside
check_window_containing, so there's no need to set it again.

Am I missing something or did this tricky code dupe valgrind?

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Eli Zaretskii]

> From: Chong Yidong <cyd@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "Stephen J. Turnbull" <stephen@xemacs.org>,  emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 23:17:19 +0800
> 
> Dan Nicolaescu <dann@gnu.org> writes:
> 
> > This is because window_from_coordinates in note_mouse_highlight is
> > passed "&part", where "part" is an uninitialized local variable.
> > When !NILP(window) "part" is not touched, so it stays uninitialized.
> > Can someone who understands this code fix this please?
> 
> Fixed; thanks.

I don't understand the fix.

--- src/xdisp.c	2011-10-19 11:46:17 +0000
+++ src/xdisp.c	2011-10-28 15:17:09 +0000
@@ -26835,11 +26835,14 @@ note_mouse_highlight (struct frame *f, i
   /* Which window is that in?  */
   window = window_from_coordinates (f, x, y, &part, 1);
 
-  /* If we were displaying active text in another window, clear that.
-     Also clear if we move out of text area in same window.  */
+  /* If displaying active text in another window, clear that.  */
   if (! EQ (window, hlinfo->mouse_face_window)
-      || (part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
-	  && !NILP (hlinfo->mouse_face_window)))
+      /* Also clear if we move out of text area in same window.  */
+      || (!NILP (hlinfo->mouse_face_window)
+	  && !NILP (window)
+	  && part != ON_TEXT
+	  && part != ON_MODE_LINE
+	  && part != ON_HEADER_LINE))
     clear_mouse_face (hlinfo);

What is the purpose of testing that "window" is non-nil?  How can it
be nil if it is EQ to hlinfo->mouse_face_window, which is non-nil?

That's the only real change in this revision, AFAICS.

What am I missing?

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dan Nicolaescu <dann@gnu.org>
>> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>,  emacs-devel@gnu.org
>> Date: Fri, 28 Oct 2011 10:38:03 -0400
>> 
>> > Valgrind produces these warnings, maybe some of them are interesting:
>> >
>> > ==30382== Conditional jump or move depends on uninitialised value(s)
>> > ==30382==    at 0x4731E6: note_mouse_highlight (xdisp.c:26841)
>> > ==30382==    by 0x4F8682: note_mouse_movement (xterm.c:3830)
>> > ==30382==    by 0x4FCB2E: handle_one_xevent (xterm.c:6652)
>> > ==30382==    by 0x4FD792: XTread_socket (xterm.c:7133)
>> > ==30382==    by 0x5437F7: read_avail_input (keyboard.c:6821)
>> > ==30382==    by 0x54409F: handle_async_input (keyboard.c:7149)
>> > ==30382==    by 0x5440BE: process_pending_signals (keyboard.c:7165)
>> > ==30382==    by 0x62D713: wait_reading_process_output (process.c:4340)
>> > ==30382==    by 0x419ED2: sit_for (dispnew.c:5971)
>> > ==30382==    by 0x53AF87: read_char (keyboard.c:2687)
>> > ==30382==    by 0x548876: read_key_sequence (keyboard.c:9290)
>> > ==30382==    by 0x538709: command_loop_1 (keyboard.c:1447)
>> 
>> This is because window_from_coordinates in note_mouse_highlight is
>> passed "&part", where "part" is an uninitialized local variable.
>> When !NILP(window) "part" is not touched, so it stays uninitialized.
>
> That's not my reading of the code.  Here's what happens in
> window_from_coordinates:
>
>   window = Qnil;
>   cw.window = &window, cw.x = x, cw.y = y; cw.part = part;
>   foreach_window (f, check_window_containing, &cw);
>
> So cw.part now holds "part", which is a pointer.
> check_window_containing does this:
>
>   found = coordinates_in_window (w, cw->x, cw->y);
>   if (found != ON_NOTHING)
>     {
>       *cw->part = found;
>       XSETWINDOW (*cw->window, w);
>
> So if it finds a window, it touches "part" through cw->part and sets
> `window' (through cw->window) to a non-nil value.  Now back in
> window_from_coordinates:
>
>   if (NILP (window)
>       && tool_bar_p
>       && WINDOWP (f->tool_bar_window)
>       && WINDOW_TOTAL_LINES (XWINDOW (f->tool_bar_window)) > 0
>       && (coordinates_in_window (XWINDOW (f->tool_bar_window), x, y)
> 	  != ON_NOTHING))
>     {
>       *part = ON_TEXT;
>       window = f->tool_bar_window;
>     }
>
> If `window' is non-nil, "part" was already set inside
> check_window_containing, so there's no need to set it again.
>
> Am I missing something or did this tricky code dupe valgrind?

I don't know anything about this code... 
A quick hack to initialize part with -42 and then abort if it's still -42 before
 "part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE"
does indeed call abort... 

Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]

[Nix]

On 28 Oct 2011, Ted Zlatanov said:
> Nix, Carsten, Sven, could you give a recent build a try?

Running 106207, started yesterday, used only for Gnus, I see:

Garbage collection stats:
((1750068 . 1022571) (81061 . 289) (17721 . 11917) 87140622 1911189 (1639 . 5992) (48873 . 14329) (459662 . 153301))

 =>	28001088+16361136 bytes in cons cells
	3890928+13872 bytes in symbols
	708840+476680 bytes in markers
	26224+95872 bytes in floats
	2736888+802424 bytes in intervals
	14709184+4905632 bytes in string headers
	87140622 bytes of string chars
	87140622 bytes of vector slots

Total bytes in lisp objects: 161780579 (live 139124963, dead 22655616)

Buffer ralloc memory usage:
107 buffers
8877238 bytes total (48855 in gaps)

USER       RSS    VSZ
nix      335476 627048

0000000001e19000 312492K rw---    [ anon ]
00007f3ab020e000      4K rw---    [ anon ]
00007f3ab4000000    132K rw---    [ anon ]
00007f3ab4021000  65404K -----    [ anon ]
00007f3ab839a000      4K rw---    [ anon ]
00007f3ab8aec000     56K rw---    [ anon ]
00007f3ab92c6000   1292K rw---    [ anon ]
00007f3ab940c000      4K rw---    [ anon ]
00007f3abac9c000      4K -----    [ anon ]
00007f3abac9d000  32768K rw---    [ anon ]


157988Kb + 8669Kb <<< 312492Kb (definitely the main heap), and this has
gone up by >150Mb since steady-state usage began yesterday.

The leak is still there, it seems.


I'll run it under valgrind and see what I can see.

-- 
NULL && (void)

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@gnu.org>
> Cc: stephen@xemacs.org,  emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 12:34:07 -0400
> 
> A quick hack to initialize part with -42 and then abort if it's still -42 before
>  "part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE"
> does indeed call abort... 

That's okay because window_from_coordinates is explicitly documented
to not touch "part" if the the position is not on any window.  Can you
show me the exact code that you used for the above test?

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Eli Zaretskii]

> Date: Fri, 28 Oct 2011 20:36:58 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: stephen@xemacs.org, emacs-devel@gnu.org
> 
> > From: Dan Nicolaescu <dann@gnu.org>
> > Cc: stephen@xemacs.org,  emacs-devel@gnu.org
> > Date: Fri, 28 Oct 2011 12:34:07 -0400
> > 
> > A quick hack to initialize part with -42 and then abort if it's still -42 before
> >  "part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE"
> > does indeed call abort... 
> 
> That's okay because window_from_coordinates is explicitly documented
> to not touch "part" if the the position is not on any window.  Can you
> show me the exact code that you used for the above test?

I fixed the problem by having "part" initialized inside
note_mouse_highlight to begin with.

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dan Nicolaescu <dann@gnu.org>
>> Cc: stephen@xemacs.org,  emacs-devel@gnu.org
>> Date: Fri, 28 Oct 2011 12:34:07 -0400
>> 
>> A quick hack to initialize part with -42 and then abort if it's still -42 before
>>  "part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE"
>> does indeed call abort... 
>
> That's okay because window_from_coordinates is explicitly documented
> to not touch "part" if the the position is not on any window.  Can you
> show me the exact code that you used for the above test?

=== modified file 'src/xdisp.c'
--- src/xdisp.c   2011-10-19 11:46:17 +0000
+++ src/xdisp.c   2011-10-28 16:29:15 +0000
@@ -26801,7 +26801,7 @@ void
 note_mouse_highlight (struct frame *f, int x, int y)
 {
   Mouse_HLInfo *hlinfo = MOUSE_HL_INFO (f);
-  enum window_part part;
+  enum window_part part = -42;
   Lisp_Object window;
   struct window *w;
   Cursor cursor = No_Cursor;
@@ -26838,7 +26838,7 @@ note_mouse_highlight (struct frame *f, i
   /* If we were displaying active text in another window, clear that.
      Also clear if we move out of text area in same window.  */
   if (! EQ (window, hlinfo->mouse_face_window)
-      || (part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
+      || ((part == -42 ? exit (1),1: 1) &&  part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
         && !NILP (hlinfo->mouse_face_window)))
     clear_mouse_face (hlinfo);
 

This is before Yidong's fix.

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@gnu.org>
> Cc: stephen@xemacs.org,  emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 14:52:35 -0400
> 
> === modified file 'src/xdisp.c'
> --- src/xdisp.c   2011-10-19 11:46:17 +0000
> +++ src/xdisp.c   2011-10-28 16:29:15 +0000
> @@ -26801,7 +26801,7 @@ void
>  note_mouse_highlight (struct frame *f, int x, int y)
>  {
>    Mouse_HLInfo *hlinfo = MOUSE_HL_INFO (f);
> -  enum window_part part;
> +  enum window_part part = -42;
>    Lisp_Object window;
>    struct window *w;
>    Cursor cursor = No_Cursor;
> @@ -26838,7 +26838,7 @@ note_mouse_highlight (struct frame *f, i
>    /* If we were displaying active text in another window, clear that.
>       Also clear if we move out of text area in same window.  */
>    if (! EQ (window, hlinfo->mouse_face_window)
> -      || (part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
> +      || ((part == -42 ? exit (1),1: 1) &&  part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
>          && !NILP (hlinfo->mouse_face_window)))
>      clear_mouse_face (hlinfo);
>  
> 
> This is before Yidong's fix.

Ah, now I understand.  The call to `exit' should have been where
there's a call to clear_mouse_face, to see if the uninitialized value
really matters, because the `!NILP (hlinfo->mouse_face_window)' part
would have taken care of the -42 value (when `part' is not touched by
window_from_coordinates, `window' is nil, so you only get to test
`part' when hlinfo->mouse_face_window is also nil).

Anyway, the variable is now initialized.

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

Dan Nicolaescu <dann@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> So I imagine one needs to do this (100% untested):
>>
>>   ./temacs -l loadup
>>   M-x load-file RET ~/.emacs RET
>
> Actually
>  ./temacs 
>
> is enough.
>
> On Fedora 15 with valgrind-3.6.1 it is possible to run 
>
> valgrind ./temacs
>
> Valgrind produces these warnings, maybe some of them are interesting:
>
> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382==    at 0x499251: coding_set_destination (coding.c:1046)
> ==30382==    by 0x4B477A: decode_coding (coding.c:7022)
> ==30382==    by 0x4B7317: decode_coding_object (coding.c:7671)
> ==30382==    by 0x63610F: Fcall_process (callproc.c:813)
> ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382==    by 0x5D5ECA: Fapply (eval.c:2479)
> ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)

This looks like setup_coding_system (or whatever else initializes struct
coding_system) does not initialize the src_pos field.

>
> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382==    at 0x6302A9: send_process (process.c:5398)
> ==30382==    by 0x630DC8: Fprocess_send_string (process.c:5648)
> ==30382==    by 0x5D6D61: Ffuncall (eval.c:2977)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)

similarly the struct coding_system.src_multibyte needs to be
initialized.


> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382==    at 0x636022: Fcall_process (callproc.c:799)
> ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382==    by 0x5D5ABB: Fapply (eval.c:2422)
> ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> ==30382==    by 0x5D5ABB: Fapply (eval.c:2422)
> ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)

similarly the struct coding_system.dst_multibyte needs to be
initialized.


> ==30382== Use of uninitialised value of size 8
> ==30382==    at 0x5B7000: mark_object (alloc.c:5674)
> ==30382==    by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
> ==30382==    by 0x5B4FE6: mark_memory (alloc.c:4274)
> ==30382==    by 0x5B507E: mark_stack (alloc.c:4532)
> ==30382==    by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
> ==30382==    by 0x5D69F0: Ffuncall (eval.c:2911)
> ==30382==    by 0x5D64CA: call2 (eval.c:2758)
> ==30382==    by 0x5809AA: Finsert_file_contents (fileio.c:4144)
> ==30382==    by 0x5D6E00: Ffuncall (eval.c:2990)
> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)

It would be interesting to know if the gc warnings are relevant...

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Chong Yidong]

Dan Nicolaescu <dann@gnu.org> writes:

>> ==30382== Use of uninitialised value of size 8
>> ==30382==    at 0x5B7000: mark_object (alloc.c:5674)
>> ==30382==    by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
>> ==30382==    by 0x5B4FE6: mark_memory (alloc.c:4274)
>> ==30382==    by 0x5B507E: mark_stack (alloc.c:4532)
>> ==30382==    by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
>> ==30382==    by 0x5D69F0: Ffuncall (eval.c:2911)
>> ==30382==    by 0x5D64CA: call2 (eval.c:2758)
>> ==30382==    by 0x5809AA: Finsert_file_contents (fileio.c:4144)
>> ==30382==    by 0x5D6E00: Ffuncall (eval.c:2990)
>> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
>> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
>> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
>
> It would be interesting to know if the gc warnings are relevant...

This is complaining about the plist member of struct Lisp_Overlay, but I
don't see how an uninitalized value could creep in there.  Overlays are
only allocated in copy_overlays and Fmake_overlay, and in both places
the plist member appears to be assigned correctly.

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Andreas Schwab]

Chong Yidong <cyd@gnu.org> writes:

> Dan Nicolaescu <dann@gnu.org> writes:
>
>>> ==30382== Use of uninitialised value of size 8
>>> ==30382==    at 0x5B7000: mark_object (alloc.c:5674)
>>> ==30382==    by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
>>> ==30382==    by 0x5B4FE6: mark_memory (alloc.c:4274)
>>> ==30382==    by 0x5B507E: mark_stack (alloc.c:4532)
>>> ==30382==    by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
>>> ==30382==    by 0x5D69F0: Ffuncall (eval.c:2911)
>>> ==30382==    by 0x5D64CA: call2 (eval.c:2758)
>>> ==30382==    by 0x5809AA: Finsert_file_contents (fileio.c:4144)
>>> ==30382==    by 0x5D6E00: Ffuncall (eval.c:2990)
>>> ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
>>> ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
>>> ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
>>
>> It would be interesting to know if the gc warnings are relevant...
>
> This is complaining about the plist member of struct Lisp_Overlay, but I
> don't see how an uninitalized value could creep in there.  Overlays are
> only allocated in copy_overlays and Fmake_overlay, and in both places
> the plist member appears to be assigned correctly.

This is while scanning the stack, where it is apparently possible that
this happens to look like a Lisp_Object, but isn't.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 16:20:25 -0400
> 
> > Valgrind produces these warnings, maybe some of them are interesting:
> >
> > ==30382== Conditional jump or move depends on uninitialised value(s)
> > ==30382==    at 0x499251: coding_set_destination (coding.c:1046)
> > ==30382==    by 0x4B477A: decode_coding (coding.c:7022)
> > ==30382==    by 0x4B7317: decode_coding_object (coding.c:7671)
> > ==30382==    by 0x63610F: Fcall_process (callproc.c:813)
> > ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382==    by 0x5D5ECA: Fapply (eval.c:2479)
> > ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> > ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> 
> This looks like setup_coding_system (or whatever else initializes struct
> coding_system) does not initialize the src_pos field.
> 
> >
> > ==30382== Conditional jump or move depends on uninitialised value(s)
> > ==30382==    at 0x6302A9: send_process (process.c:5398)
> > ==30382==    by 0x630DC8: Fprocess_send_string (process.c:5648)
> > ==30382==    by 0x5D6D61: Ffuncall (eval.c:2977)
> > ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> > ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> > ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> 
> similarly the struct coding_system.src_multibyte needs to be
> initialized.
> 
> 
> > ==30382== Conditional jump or move depends on uninitialised value(s)
> > ==30382==    at 0x636022: Fcall_process (callproc.c:799)
> > ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382==    by 0x5D5ABB: Fapply (eval.c:2422)
> > ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> > ==30382==    by 0x5D5ABB: Fapply (eval.c:2422)
> > ==30382==    by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> 
> similarly the struct coding_system.dst_multibyte needs to be
> initialized.
> 
> 
> > ==30382== Use of uninitialised value of size 8
> > ==30382==    at 0x5B7000: mark_object (alloc.c:5674)
> > ==30382==    by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
> > ==30382==    by 0x5B4FE6: mark_memory (alloc.c:4274)
> > ==30382==    by 0x5B507E: mark_stack (alloc.c:4532)
> > ==30382==    by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
> > ==30382==    by 0x5D69F0: Ffuncall (eval.c:2911)
> > ==30382==    by 0x5D64CA: call2 (eval.c:2758)
> > ==30382==    by 0x5809AA: Finsert_file_contents (fileio.c:4144)
> > ==30382==    by 0x5D6E00: Ffuncall (eval.c:2990)
> > ==30382==    by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382==    by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382==    by 0x5D6F4C: Ffuncall (eval.c:3023)
> 
> It would be interesting to know if the gc warnings are relevant...

I suggest to file a separate bug report about each valgrind warning.

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Stefan Monnier]

> This is while scanning the stack, where it is apparently possible that
> this happens to look like a Lisp_Object, but isn't.

That doesn't explain it, tho: the conservative stack scanning does check
to make sure the suspected pointer really does point to a real (tho
possibly dead) object, and we always know its type.

I.e. barring bugs in the conservative-scanning code, it should only
cause dead objects to be kept around (and scanned).  But those used to
be live, so they have been initialized.


        Stefan

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> This is while scanning the stack, where it is apparently possible that
>> this happens to look like a Lisp_Object, but isn't.
>
> That doesn't explain it, tho: the conservative stack scanning does check
> to make sure the suspected pointer really does point to a real (tho
> possibly dead) object, and we always know its type.

It turns out there's a --track-origins=yes valgrind flag.  With that
turned on we get this warning:

==9622== Conditional jump or move depends on uninitialised value(s)
==9622==    at 0x5B3D89: mem_find (alloc.c:3521)
==9622==    by 0x5B4AEB: mark_maybe_object (alloc.c:4103)
==9622==    by 0x5B50E6: mark_memory (alloc.c:4275)
==9622==    by 0x5B517E: mark_stack (alloc.c:4533)
==9622==    by 0x5B6048: Fgarbage_collect (alloc.c:5120)
==9622==    by 0x5D542C: eval_sub (eval.c:2217)
==9622==    by 0x602A0B: readevalloop (lread.c:1838)
==9622==    by 0x6015B6: Fload (lread.c:1316)
==9622==    by 0x5D59FF: eval_sub (eval.c:2336)
==9622==    by 0x602A0B: readevalloop (lread.c:1838)
==9622==    by 0x6015B6: Fload (lread.c:1316)
==9622==    by 0x5D59FF: eval_sub (eval.c:2336)
==9622==  Uninitialised value was created by a stack allocation
==9622==    at 0x5B5D00: Fgarbage_collect (alloc.c:4993)
==9622== 

And indeed, there's:
  Lisp_Object total[8];
in Fgarbage_collect.
What should we do about this?

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Andreas Schwab]

Dan Nicolaescu <dann@gnu.org> writes:

> What should we do about this?

It is obvious that the stack scanning can find all kinds of stale data,
but that is inherent to conservative gc.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: valgrind warnings

[Paul Eggert]

On 11/04/11 05:51, Dan Nicolaescu wrote:
> What should we do about this?

There's nothing wrong with the code; this is a false
alarm of valgrind, which is inherent to the way a
conservative garbage collector works.

One way to work around this valgrind issue is to supply
a suppression file, e.g., put something like the
following into a new file emacs/src/valgrind.supp:

{
   Conservative garbage collection Cond
   Memcheck:Cond
   ...
   fun:mark_maybe_*
}
{
   Conservative garbage collection Value8
   Memcheck:Value8
   ...
   fun:mark_maybe_*
}

(this is just an indication of what it would look
like; more is needed).  The goal is to suppress
false positives while minimizing false negatives.

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Stefan Monnier]

>>> This is while scanning the stack, where it is apparently possible that
>>> this happens to look like a Lisp_Object, but isn't.
>> That doesn't explain it, tho: the conservative stack scanning does check
>> to make sure the suspected pointer really does point to a real (tho
>> possibly dead) object, and we always know its type.
> It turns out there's a --track-origins=yes valgrind flag.  With that
> turned on we get this warning:

> ==9622== Conditional jump or move depends on uninitialised value(s)
> ==9622==    at 0x5B3D89: mem_find (alloc.c:3521)
> ==9622==    by 0x5B4AEB: mark_maybe_object (alloc.c:4103)
> ==9622==    by 0x5B50E6: mark_memory (alloc.c:4275)
> ==9622==    by 0x5B517E: mark_stack (alloc.c:4533)
> ==9622==    by 0x5B6048: Fgarbage_collect (alloc.c:5120)
> ==9622==    by 0x5D542C: eval_sub (eval.c:2217)
> ==9622==    by 0x602A0B: readevalloop (lread.c:1838)
> ==9622==    by 0x6015B6: Fload (lread.c:1316)
> ==9622==    by 0x5D59FF: eval_sub (eval.c:2336)
> ==9622==    by 0x602A0B: readevalloop (lread.c:1838)
> ==9622==    by 0x6015B6: Fload (lread.c:1316)
> ==9622==    by 0x5D59FF: eval_sub (eval.c:2336)
> ==9622==  Uninitialised value was created by a stack allocation
> ==9622==    at 0x5B5D00: Fgarbage_collect (alloc.c:4993)
> ==9622== 

> And indeed, there's:
>   Lisp_Object total[8];
> in Fgarbage_collect.

Ah, right, so the problem is not that the stack "points" to some
uninitialized value, but that the stack itself contains uninitialized
values.  Yes, that's perfectly harmless (at least no worse than the
arbitrary initialized values we can find).

> What should we do about this?

Tell valgrind that it's a non-issue.


        Stefan

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> This is while scanning the stack, where it is apparently possible that
>>>> this happens to look like a Lisp_Object, but isn't.
>>> That doesn't explain it, tho: the conservative stack scanning does check
>>> to make sure the suspected pointer really does point to a real (tho
>>> possibly dead) object, and we always know its type.
>> It turns out there's a --track-origins=yes valgrind flag.  With that
>> turned on we get this warning:
>
>> ==9622== Conditional jump or move depends on uninitialised value(s)
>> ==9622==    at 0x5B3D89: mem_find (alloc.c:3521)
>> ==9622==    by 0x5B4AEB: mark_maybe_object (alloc.c:4103)
>> ==9622==    by 0x5B50E6: mark_memory (alloc.c:4275)
>> ==9622==    by 0x5B517E: mark_stack (alloc.c:4533)
>> ==9622==    by 0x5B6048: Fgarbage_collect (alloc.c:5120)
>> ==9622==    by 0x5D542C: eval_sub (eval.c:2217)
>> ==9622==    by 0x602A0B: readevalloop (lread.c:1838)
>> ==9622==    by 0x6015B6: Fload (lread.c:1316)
>> ==9622==    by 0x5D59FF: eval_sub (eval.c:2336)
>> ==9622==    by 0x602A0B: readevalloop (lread.c:1838)
>> ==9622==    by 0x6015B6: Fload (lread.c:1316)
>> ==9622==    by 0x5D59FF: eval_sub (eval.c:2336)
>> ==9622==  Uninitialised value was created by a stack allocation
>> ==9622==    at 0x5B5D00: Fgarbage_collect (alloc.c:4993)
>> ==9622== 
>
>> And indeed, there's:
>>   Lisp_Object total[8];
>> in Fgarbage_collect.
>
> Ah, right, so the problem is not that the stack "points" to some
> uninitialized value, but that the stack itself contains uninitialized
> values.  Yes, that's perfectly harmless (at least no worse than the
> arbitrary initialized values we can find).
>
>> What should we do about this?
>
> Tell valgrind that it's a non-issue.

There is a way to do that, write a configuration file and pass it to
valgrind with:  --suppressions=FILENAME

Such a file is attached.

I think it should be checked in src/ so that everyone can make use of it
and improve it.  OK to check in?


[-- Attachment #2: valgrind.supp --]
[-- Type: text/plain, Size: 1242 bytes --]

# valgrind supression file
# generated on a x86_64-unknown-linux-gnu machine
# Usage:
#    valgrind --suppressions=valgrind.supp ./temacs

# This looks like this:
# Invalid read of size 4
#    at 0x511B3C: init_buffer (buffer.c:5037)
#    by 0x40E293: main (emacs.c:1452)
#  Address 0x50cff24 is 68 bytes inside a block of size 70 alloc'd
#    at 0x4A06582: realloc (vg_replace_malloc.c:525)
#    by 0x511B23: init_buffer (buffer.c:5071)
#    by 0x40E293: main (emacs.c:1452)
#
# It is due to the optimized strlen implementation that loads 32 bits at a time.
{
   optimized_strlen_in_init_buffer
   Memcheck:Addr4
   fun:init_buffer
   fun:main
}

# Disable various warnings due to garbage collection scanning the stack.
{
   <GCcond>
   Memcheck:Cond
   fun:Fgarbage_collect
   fun:*
}
{
   <GCvalue>
   Memcheck:Value8
   fun:Fgarbage_collect
   fun:*
}
{
   <mark>
   Memcheck:Value8
   fun:mark_object
   fun:*
}
{
   mark_object_cond
   Memcheck:Cond
   fun:mark_object
   fun:*
}
{
   mark_vectorlike_value8
   Memcheck:Value8
   fun:mark_vectorlike
   fun:Fgarbage_collect
   fun:*
}
{
   survives_gc_p_cond
   Memcheck:Cond
   fun:survives_gc_p
   fun:sweep_weak_table
   fun:sweep_weak_hash_tables
   fun:Fgarbage_collect
   fun:*
}

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

I like the idea of creating a src/valgrind.supp file.
I looked at Dan's proposal and merged it with mine,
adding some comments to try to explain things
(merged version attached).

One question/problem that I had with Dan's is this rule:

  {
    survives_gc_p_cond
    Memcheck:Cond
    fun:survives_gc_p
    fun:sweep_weak_table
    fun:sweep_weak_hash_tables
    fun:Fgarbage_collect
    fun:*
  }

As near as I can make out, if survives_gc_p is accessing
uninitialized storage, it's buggy, no?  Shouldn't we omit
this rule and fix the bug instead?

[-- Attachment #2: valgrind.supp --]
[-- Type: text/plain, Size: 1408 bytes --]

# valgrind suppression file
# Usage:
#    valgrind --suppressions=valgrind.supp ./temacs

# Conservative garbage collection inherently looks at uninitialized values,
# and mark_maybe_object is at the core of this.
{
   mark_maybe_object Cond - Conservative garbage collection
   Memcheck:Cond
   fun:mark_maybe_object
}
{
   mark_maybe_object Value8 - Conservative garbage collection
   Memcheck:Value8
   fun:mark_maybe_object
}
# mark_maybe_object can be inlined from Fgarbage_collect.
{
   mark_maybe_object Cond - Conservative garbage collection
   Memcheck:Cond
   fun:Fgarbage_collect
}
{
   mark_maybe_object Value8 - Conservative garbage collection
   Memcheck:Value8
   fun:Fgarbage_collect
}
# mark_maybe_object can be inlined from mark_object.
{
   mark_maybe_object Cond - Conservative garbage collection
   Memcheck:Cond
   fun:mark_object
}
{
   mark_maybe_object Value8 - Conservative garbage collection
   Memcheck:Value8
   fun:mark_object
}
# mark_maybe_object can be inlined from mark_vectorlike.
{
   mark_vectorlike Cond - Conservative garbage collection
   Memcheck:Cond
   fun:mark_vectorlike
}
{
   mark_vectorlike Value8 - Conservative garbage collection
   Memcheck:Value8
   fun:mark_vectorlike
}

# On one circa-2011 x86-64 GNU/Linux platform, strlen is inlined to
# something that loads 4 bytes at a time.
{
   init_buffer optimized strlen
   Memcheck:Addr4
   fun:init_buffer
}

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

Paul Eggert <eggert@cs.ucla.edu> writes:

> I like the idea of creating a src/valgrind.supp file.
> I looked at Dan's proposal and merged it with mine,
> adding some comments to try to explain things
> (merged version attached).

This version looks nicer!

> One question/problem that I had with Dan's is this rule:
>
>   {
>     survives_gc_p_cond
>     Memcheck:Cond
>     fun:survives_gc_p
>     fun:sweep_weak_table
>     fun:sweep_weak_hash_tables
>     fun:Fgarbage_collect
>     fun:*
>   }
>
> As near as I can make out, if survives_gc_p is accessing
> uninitialized storage, it's buggy, no?  Shouldn't we omit
> this rule and fix the bug instead?

The full warning looks like this (when also using --track-origins=yes):

==4937== Conditional jump or move depends on uninitialised value(s)
==4937==    at 0x5B7567: survives_gc_p (alloc.c:5838)
==4937==    by 0x5E5FBD: sweep_weak_table (fns.c:3965)
==4937==    by 0x5E62DE: sweep_weak_hash_tables (fns.c:4066)
==4937==    by 0x5B75C8: gc_sweep (alloc.c:5850)
==4937==    by 0x5B6195: Fgarbage_collect (alloc.c:5203)
==4937==    by 0x62301A: exec_byte_code (bytecode.c:820)
==4937==    by 0x5D7A96: funcall_lambda (eval.c:3205)
==4937==    by 0x5D754A: apply_lambda (eval.c:3082)
==4937==    by 0x5D5B4C: eval_sub (eval.c:2367)
==4937==    by 0x5D3AF6: internal_lisp_condition_case (eval.c:1453)
==4937==    by 0x623C06: exec_byte_code (bytecode.c:981)
==4937==    by 0x5D7A96: funcall_lambda (eval.c:3205)
==4937==  Uninitialised value was created by a stack allocation
==4937==    at 0x6224AA: exec_byte_code (bytecode.c:437)
==4937== 

Do you think that's a problem?

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Paul Eggert]

On 11/05/11 15:08, Dan Nicolaescu wrote:
> Do you think that's a problem?

It looks suspicious, yes.

I don't have that problem with bzr 106302
(Fedora 15 x86-64, Emacs compiled with GCC 4.6.2).
I do recall seeing it in earlier bzrs on the trunk, though.

What happens when you run:

valgrind --db-attach=yes --suppressions=valgrind.supp ./temacs

Can you use GDB to navigate to the offending uninitialized value
and see why the garbage collector is trying to look
at that stack location?

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Dan Nicolaescu]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 11/05/11 15:08, Dan Nicolaescu wrote:
>> Do you think that's a problem?
>
> It looks suspicious, yes.
>
> I don't have that problem with bzr 106302
> (Fedora 15 x86-64, Emacs compiled with GCC 4.6.2).
> I do recall seeing it in earlier bzrs on the trunk, though.

I use the same platform, but the compiler is
gcc (GCC) 4.6.1 20110908 (Red Hat 4.6.1-9)
I compile with -g -O0.


> What happens when you run:
>
> valgrind --db-attach=yes --suppressions=valgrind.supp ./temacs
>
> Can you use GDB to navigate to the offending uninitialized value
> and see why the garbage collector is trying to look
> at that stack location?

Tried that, but I didn't find anything that made sense...

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 267 bytes --]

On 11/06/11 04:37, Dan Nicolaescu wrote:

> Tried that, but I didn't find anything that made sense...

Valgrind isn't always supposed to make sense :-)

How about the attached valgrind.supp instead?
It suppresses more diagnostics, and with luck it'll suppress yours.

[-- Attachment #2: valgrind.supp --]
[-- Type: text/plain, Size: 1337 bytes --]

# valgrind suppression file
# Usage:
#    valgrind --suppressions=valgrind.supp ./temacs

# Conservative garbage collection inherently looks at uninitialized values,
# and Fgarbage_collect and its callees all depend on this.
# It's hard to separate out exactly which callees need to be listed here,
# since the C compiler can inline them.  Also, valgrind doesn't care
# about the use of uninitialized variables directly, only when their values
# are eventually used.  So just list Fgarbage_collect and its callees.
{
   Fgarbage_collect Cond - conservative garbage collection
   Memcheck:Cond
   ...
   fun:Fgarbage_collect
}
{
   Fgarbage_collect Value8 - conservative garbage collection
   Memcheck:Value8
   ...
   fun:Fgarbage_collect
}
# valgrind only looks at the last few callees on the stack, but
# mark_object can call itself recursively and deeply.  So list
# it too, in case Fgarbage_collect is a long way from the stack top.
{
   Fgarbage_collect Cond - conservative garbage collection
   Memcheck:Cond
   ...
   fun:mark_object
}
{
   Fgarbage_collect Value8 - conservative garbage collection
   Memcheck:Value8
   ...
   fun:mark_object
}

# On one circa-2011 x86-64 GNU/Linux platform, strlen is inlined to
# something that loads 4 bytes at a time.
{
   init_buffer optimized strlen
   Memcheck:Addr4
   fun:init_buffer
}

Re: valgrind warnings [Re: Emacs bzr memory footprint]

[Florian Weimer]

* Ted Zlatanov:

> On Fri, 28 Oct 2011 07:52:12 -0400 Dan Nicolaescu <dann@gnu.org> wrote: 
> DN> On Fedora 15 with valgrind-3.6.1 it is possible to run 
>
> DN> valgrind ./temacs
>
> I ran this and opened and closed a GnuTLS connection many times:
>
> (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
>
> I did not see any errors printed when I did this, but on exit a lot of
> information was printed that did not seem related (I could not analyze
> it all, it was too much data and I don't know what to look for other
> than "gnutls").  Is this enough to tell us there are no memory leaks in
> the "open connection" part of the code?  I'll test sending and receiving
> data if so.

I think you need to run

valgrind --leak-check=yes ./temacs

and actually terminate the program to get the dump of leaked
resources.

Has the bug been identified in the meantime?  I've got some very
long-running processes using GnuTLS which grow without bounds,
too---but I've been blaming this on bugs in my Perl wrapper.  In that
cases, just opening and closing connections (and sending a bit of
data) does not trigger the leak condition, either.