* Re: Emacs bzr memory footprint
2011-10-21 13:30 ` Stefan Monnier
@ 2011-10-21 15:02 ` Nix
2011-10-21 17:52 ` Stefan Monnier
2011-10-21 15:34 ` Dmitry Antipov
2011-10-25 17:35 ` Nix
2 siblings, 1 reply; 113+ messages in thread
From: Nix @ 2011-10-21 15:02 UTC (permalink / raw)
To: Stefan Monnier; +Cc: John Wiegley, emacs-devel
On 21 Oct 2011, Stefan Monnier said:
>> STIME RSS VSZ
>> Oct07 876524 1170572
>> Oct07 227016 499588
>
> I've just installed memory-usage.el as a package in the `elpa' branch
> (you'll soon be able to install it using package.el and in the mean time
> you can download it from http://bzr.savannah.gnu.org/lh/emacs/elpa/download/head:/memoryusage.el-20111021130523-o1v7kfuzrat3pcd9-2/memory-usage.el).
>
> It basically looks at the garbage-collect output and the various
> buffer's size to provide you with a human-readable description of the
> memory in use, from Elisp's point of view. Don't expect too much of it,
> but it can be helpful.
>
> Could you try it out?
I am actually generating a gigantic amount of debugging output at GC
time using a patch from Dmitry Antipov right now: in a few days once
it's bloated up we'll see what we can see.
I'll try memory-usage.el out too once the bloating really kicks in:
I was looking for it earlier but then remembered that it's XEmacs that
has 'memory-usage-stats' as an interactive subr :)
> If this output does not explain the process's size you're seeing, then
> we have a leak in the C code somewhere. If it does, then we have either
> a leak in Elisp, or at least an excessive memory use by some package,
> and hopefully we can at least figure out which category of object
> is involved.
... or the same thing XEmacs apparently had, which was excessive memory
consumption by one or more other still-unidentified C libraries.
>> Unfortunately that means, sigh, using our own malloc() again, which is
>> probably more painful than useful.
>
> Emacs already uses its own malloc for most objects: it uses malloc
> directly to allocate vectors (and this is actually something that we
> might want to change, because it (indirectly) comes with an O(log n)
> cost), but for strings, conses, markers, overlays, and floats it uses
> malloc only to get a chunk of memory which it then manages on its own.
True! So it wouldn't be as hellish to write our own allocator as it
would be for most projects, because we've done 90% of the work already
(including the hard part: allocating things of variable size).
> hash-tables, and a few more; and "markers" similarly include a few
> non-marker objects, mostly overlays. Oh, and "intervals" are objects
> used to store text-properties.
Yeah. As an aside: most of the inside of Emacs is quite nice, but I am
seriously unimpressed by overlay storage. Linear scans for
'overlays-at'? Ew. One of these years I might try to implement something
better, like what XEmacs did for extents, so you can find extents over a
given buffer position more efficiently... (not because this is visibly a
problem in profiles that I've seen, merely because the inefficiency
offends me :) )
>> I suspect actually proving my contention first would be a good
>> idea. Not sure how to get the addresses of Lisp objects from a running
>> Emacs though: gdb, presumably.
>
> I'd hack src/alloc.c to export the needed info to Elisp. But maybe
> memory-usage will already give us enough info that this won't
> be necessary.
Already done by Dmitry before I could do more than blink :)
--
NULL && (void)
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 15:02 ` Nix
@ 2011-10-21 17:52 ` Stefan Monnier
2011-10-21 19:12 ` Nix
0 siblings, 1 reply; 113+ messages in thread
From: Stefan Monnier @ 2011-10-21 17:52 UTC (permalink / raw)
To: Nix; +Cc: John Wiegley, emacs-devel
>> If this output does not explain the process's size you're seeing, then
>> we have a leak in the C code somewhere. If it does, then we have either
>> a leak in Elisp, or at least an excessive memory use by some package,
>> and hopefully we can at least figure out which category of object
>> is involved.
> ... or the same thing XEmacs apparently had, which was excessive memory
> consumption by one or more other still-unidentified C libraries.
Yes, that's what I meant by "in the C code somewhere". Another thread
recently seemed to suggest that maybe the new C-level support for gnutls
may cause such problems.
> True! So it wouldn't be as hellish to write our own allocator as it
> would be for most projects, because we've done 90% of the work already
> (including the hard part: allocating things of variable size).
Actually, no, that doesn't include the hard part: we only do our own
allocation for fixed-size objects (cons, floats, markers&overlays,
symbols, string headers) plus relocatable variable size objects (string
text). For the non-relocatable, non-fixed size objects (i.e. vectors)
we defer to malloc.
> Yeah. As an aside: most of the inside of Emacs is quite nice, but I am
> seriously unimpressed by overlay storage. Linear scans for
> 'overlays-at'? Ew. One of these years I might try to implement something
> better, like what XEmacs did for extents, so you can find extents over a
> given buffer position more efficiently... (not because this is visibly a
> problem in profiles that I've seen, merely because the inefficiency
> offends me :) )
Yes, overlays (and to some extent markers) are in serious needs of
a redesign. We get bitten by this performance problem, which is why
a lot of code goes through a fair bit of trouble to avoid overlays and
use text-properties instead (and the code that doesn't bother doing
that is usually only usable on small buffers).
Stefan
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: Emacs bzr memory footprint
2011-10-21 17:52 ` Stefan Monnier
@ 2011-10-21 19:12 ` Nix
0 siblings, 0 replies; 113+ messages in thread
From: Nix @ 2011-10-21 19:12 UTC (permalink / raw)
To: Stefan Monnier; +Cc: John Wiegley, emacs-devel
On 21 Oct 2011, Stefan Monnier spake thusly:
>> True! So it wouldn't be as hellish to write our own allocator as it
>> would be for most projects, because we've done 90% of the work already
>> (including the hard part: allocating things of variable size).
>
> Actually, no, that doesn't include the hard part: we only do our own
> allocation for fixed-size objects (cons, floats, markers&overlays,
> symbols, string headers) plus relocatable variable size objects (string
> text). For the non-relocatable, non-fixed size objects (i.e. vectors)
> we defer to malloc.
Oh heck. I assumed that were were allocating the strings themselves, not
just their headers, and didn't check the source code. Mea culpa.
>> Yeah. As an aside: most of the inside of Emacs is quite nice, but I am
>> seriously unimpressed by overlay storage. Linear scans for
>> 'overlays-at'? Ew. One of these years I might try to implement something
>> better, like what XEmacs did for extents, so you can find extents over a
>> given buffer position more efficiently... (not because this is visibly a
>> problem in profiles that I've seen, merely because the inefficiency
>> offends me :) )
>
> Yes, overlays (and to some extent markers) are in serious needs of
> a redesign. We get bitten by this performance problem, which is why
> a lot of code goes through a fair bit of trouble to avoid overlays and
> use text-properties instead (and the code that doesn't bother doing
> that is usually only usable on small buffers).
Oh good. On my huge todo list it goes then!
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 13:30 ` Stefan Monnier
2011-10-21 15:02 ` Nix
@ 2011-10-21 15:34 ` Dmitry Antipov
2011-10-21 18:00 ` Stefan Monnier
2011-10-25 17:35 ` Nix
2 siblings, 1 reply; 113+ messages in thread
From: Dmitry Antipov @ 2011-10-21 15:34 UTC (permalink / raw)
To: emacs-devel
On 10/21/2011 05:30 PM, Stefan Monnier wrote:
> Emacs already uses its own malloc for most objects: it uses malloc
> directly to allocate vectors (and this is actually something that we
> might want to change, because it (indirectly) comes with an O(log n)
> cost), but for strings, conses, markers, overlays, and floats it uses
> malloc only to get a chunk of memory which it then manages on its own.
> `memory-usage' does give you this kind of info when it says
> "5743392+864696 bytes in cons cells" which means 5MB of live cons-cells,
> and 800KB of cons-cells that are free (i.e. they are in a memory chunk
> that Emacs can't return to malloc because that chunk also contains live
> cons cells).
This creates two-level fragmentation: the whole heap is fragmented as seen
by malloc(), and allocated space is fragmented too, as seen by Emacs. Since
a lot of non-lisp allocation requests are satisfied from the same sbrk()ed
heap, here is the typical scenario: at startup, there is 10M of live lisp data.
Solid workload (e.g. visit 1000 files at once) increases this, say, to 100M.
When the workload is finished (kill all file-backed 1000 buffers), live lisp data
shrinks back to 10M. The rest 90M can't be freed because a) small percentage
is interleaved with live lisp data within the chunks managed by Emacs itself
and b) the rest, being freed by Emacs, can't be freed by malloc because
freed chunks are interleaved with non-freed, and all chunks are interleaved
with freed/lived lisp vectors and freed/lived non-lisp allocations. If next
solid workload will consume 100M of the heap, the most of this 90M will be re-used -
but it will never freed.
Due to all of the above, I suspect that the mostly-copying collector, where all
small (say, < 4K) lisp (and only lisp) allocation requests are satisfied from
1M-4M areas, might have substantial advantages over plain mark&sweep scheme.
Dmitry
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 15:34 ` Dmitry Antipov
@ 2011-10-21 18:00 ` Stefan Monnier
2011-10-21 18:27 ` Óscar Fuentes
2011-10-22 7:34 ` Carsten Mattner
0 siblings, 2 replies; 113+ messages in thread
From: Stefan Monnier @ 2011-10-21 18:00 UTC (permalink / raw)
To: Dmitry Antipov; +Cc: emacs-devel
> This creates two-level fragmentation: the whole heap is fragmented as seen
> by malloc(), and allocated space is fragmented too, as seen by Emacs. Since
Yes.
> solid workload will consume 100M of the heap, the most of this 90M will be re-used -
Exactly: so this fragmentation is not a serious problem in the sense
that this free memory can be (re)used by Emacs.
> but it will never freed.
Right, so it's a problem in the sense that this memory will not be
available to other processes.
While this is not perfect, I don't consider it to be a serious problem.
E.g. when people complain about Emacs using a lot of memory it's always
(until now and I think in Nix's case as well) a case of too much memory
being used because of something like a leak, and not just "Emacs doesn't
return its free memory to the OS".
E.g. in the case of Nix, I think it's pretty clear that Emacs should
never have gotten to the point where it used up 800MB of memory.
> Due to all of the above, I suspect that the mostly-copying collector, where all
> small (say, < 4K) lisp (and only lisp) allocation requests are satisfied from
> 1M-4M areas, might have substantial advantages over plain mark&sweep scheme.
There's no argument that pretty anything else than mark&sweep is better
than mark&sweep in some significant respect.
Stefan
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: Emacs bzr memory footprint
2011-10-21 18:00 ` Stefan Monnier
@ 2011-10-21 18:27 ` Óscar Fuentes
2011-10-21 18:58 ` Eli Zaretskii
2011-10-21 19:11 ` Nix
2011-10-22 7:34 ` Carsten Mattner
1 sibling, 2 replies; 113+ messages in thread
From: Óscar Fuentes @ 2011-10-21 18:27 UTC (permalink / raw)
To: emacs-devel
Stefan Monnier <monnier@iro.umontreal.ca> writes:
[snip]
>> but it will never freed.
>
> Right, so it's a problem in the sense that this memory will not be
> available to other processes.
>
> While this is not perfect, I don't consider it to be a serious problem.
> E.g. when people complain about Emacs using a lot of memory it's always
> (until now and I think in Nix's case as well) a case of too much memory
> being used because of something like a leak, and not just "Emacs doesn't
> return its free memory to the OS".
The memory for the text of a buffer, is a single chunk? Or, more
generally, are large chunks of memory allocated for big buffers? If the
answer is yes, memory fragmentation may be problematic for those who
work with large buffers, unless the allocating algorithm avoids that.
[snip]
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 18:27 ` Óscar Fuentes
@ 2011-10-21 18:58 ` Eli Zaretskii
2011-10-21 19:11 ` Nix
1 sibling, 0 replies; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-21 18:58 UTC (permalink / raw)
To: Óscar Fuentes; +Cc: emacs-devel
> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 21 Oct 2011 20:27:28 +0200
>
> The memory for the text of a buffer, is a single chunk?
Yes.
> Or, more generally, are large chunks of memory allocated for big
> buffers?
Yes.
> If the answer is yes, memory fragmentation may be problematic for
> those who work with large buffers, unless the allocating algorithm
> avoids that.
It does.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 18:27 ` Óscar Fuentes
2011-10-21 18:58 ` Eli Zaretskii
@ 2011-10-21 19:11 ` Nix
2011-10-21 20:18 ` Eli Zaretskii
1 sibling, 1 reply; 113+ messages in thread
From: Nix @ 2011-10-21 19:11 UTC (permalink / raw)
To: Óscar Fuentes; +Cc: emacs-devel
On 21 Oct 2011, Óscar Fuentes told this:
> The memory for the text of a buffer, is a single chunk? Or, more
> generally, are large chunks of memory allocated for big buffers? If the
> answer is yes, memory fragmentation may be problematic for those who
> work with large buffers, unless the allocating algorithm avoids that.
Emacs goes to some lengths to avoid allocating memory on the brk()ed
heap for buffers of significant size. The rules are somewhat arcane.
On a limited number of platforms (currently Cygwin, FreeBSD and IRIX
6.5), mmap() is used to map buffer text directly.
On platforms using the copy of Doug Lea's malloc() in src/gmalloc.c (or
using Doug Lea's malloc without being a GNU system, if any such systems
exist), buffers are allocated using the frankly terrifying
block-relocating memory allocator in ralloc.c, which uses sbrk()
directly and arranges for blocks allocated that way to dodge out of the
way of malloc() using the morecore_hook. (I'd be scared to trust my data
to this one.)
On GNU platforms which appear to be using Doug Lea's malloc(), we turn
this off and just use malloc() directly.
Emacs relies on all of these allocating the buffer storage in a single
(virtually) contiguous chunk (modulo the gap): the gap-motion code in
insdel.c, and much else, depends on being able to use simple pointer
arithmetic to move around the buffer.
--
NULL && (void)
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 19:11 ` Nix
@ 2011-10-21 20:18 ` Eli Zaretskii
2011-10-21 20:21 ` Nix
0 siblings, 1 reply; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-21 20:18 UTC (permalink / raw)
To: Nix; +Cc: ofv, emacs-devel
> From: Nix <nix@esperi.org.uk>
> Date: Fri, 21 Oct 2011 20:11:29 +0100
> Cc: emacs-devel@gnu.org
>
> I'd be scared to trust my data to this one.
Don't be. It's well tested on many platforms, and the Windows build
still uses it. I don't think there are any bug reports related to it.
> Emacs relies on all of these allocating the buffer storage in a single
> (virtually) contiguous chunk (modulo the gap)
The buffer text is a contiguous chunk of memory; the gap is just a
portion of that chunk ready to receive inserted text.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 20:18 ` Eli Zaretskii
@ 2011-10-21 20:21 ` Nix
0 siblings, 0 replies; 113+ messages in thread
From: Nix @ 2011-10-21 20:21 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: ofv, emacs-devel
On 21 Oct 2011, Eli Zaretskii said:
>> From: Nix <nix@esperi.org.uk>
>> Date: Fri, 21 Oct 2011 20:11:29 +0100
>> Cc: emacs-devel@gnu.org
>>
>> I'd be scared to trust my data to this one.
>
> Don't be. It's well tested on many platforms, and the Windows build
> still uses it. I don't think there are any bug reports related to it.
OK. Doesn't make it less terrifying though. I don't think I've ever seen
a clearer example of duelling allocators. :)
>> Emacs relies on all of these allocating the buffer storage in a single
>> (virtually) contiguous chunk (modulo the gap)
>
> The buffer text is a contiguous chunk of memory; the gap is just a
> portion of that chunk ready to receive inserted text.
Exactly. (I think I've implemented buffer-gap-based algorithms three
separate times: they seems kludgy, but are so much simpler than pretty
much every alternative, and are fast in all but the most uncommon
cases.)
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 18:00 ` Stefan Monnier
2011-10-21 18:27 ` Óscar Fuentes
@ 2011-10-22 7:34 ` Carsten Mattner
2011-10-22 8:30 ` Stephen J. Turnbull
` (2 more replies)
1 sibling, 3 replies; 113+ messages in thread
From: Carsten Mattner @ 2011-10-22 7:34 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Dmitry Antipov, emacs-devel
On Fri, Oct 21, 2011 at 8:00 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> > This creates two-level fragmentation: the whole heap is fragmented as seen
> > by malloc(), and allocated space is fragmented too, as seen by Emacs. Since
>
> Yes.
>
> > solid workload will consume 100M of the heap, the most of this 90M will be re-used -
>
> Exactly: so this fragmentation is not a serious problem in the sense
> that this free memory can be (re)used by Emacs.
>
> > but it will never freed.
>
> Right, so it's a problem in the sense that this memory will not be
> available to other processes.
>
> While this is not perfect, I don't consider it to be a serious problem.
> E.g. when people complain about Emacs using a lot of memory it's always
> (until now and I think in Nix's case as well) a case of too much memory
> being used because of something like a leak, and not just "Emacs doesn't
> return its free memory to the OS".
It is a problem for everybody like me who doesn't have
gigs of free RAM. Sorry.
I'll remember to --disable-gnutls and not use Gnus for my
own safety, for the moment. That's a good catch and seems
to help.
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: Emacs bzr memory footprint
2011-10-22 7:34 ` Carsten Mattner
@ 2011-10-22 8:30 ` Stephen J. Turnbull
2011-10-22 12:03 ` Carsten Mattner
2011-10-22 8:44 ` Emacs bzr memory footprint Sven Joachim
2011-10-23 18:04 ` Stefan Monnier
2 siblings, 1 reply; 113+ messages in thread
From: Stephen J. Turnbull @ 2011-10-22 8:30 UTC (permalink / raw)
To: Carsten Mattner; +Cc: Dmitry Antipov, Stefan Monnier, emacs-devel
Carsten Mattner writes:
> On Fri, Oct 21, 2011 at 8:00 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
> > While this is not perfect, I don't consider it to be a serious problem.
> > E.g. when people complain about Emacs using a lot of memory it's always
> > (until now and I think in Nix's case as well) a case of too much memory
> > being used because of something like a leak, and not just "Emacs doesn't
> > return its free memory to the OS".
>
> It is a problem for everybody like me who doesn't have
> gigs of free RAM. Sorry.
Nobody is saying that running out of memory isn't a serious problem.
Stefan's point is simply that "if you don't have enough RAM to use
Emacs comfortably, you don't have enough RAM to use Emacs comfortably."
There are applications that use huge amounts of memory at
initialization or in the early part of an algorithm, then can free it
and never need it again. In those applications, freeing memory after
use helps a lot. Emacs isn't one of those; average memory usage is
normally a reasonably high fraction of peak usage (at least 75%, often
effectively 100%, in my own usage patterns) and the peaks tend to
recur. If that's more memory than you can afford, there's little we
can do. Probably memory usage can be decreased on average or even at
peak by a few percent, but is it worth the effort to develop and
maintain very sophisticated allocators? While it is my belief that
XEmacs's current excess memory consumption is related to system
libraries like X, I have to admit that reports started to appear about
the same time that a new garbage collection framework was introduced
-- it's quite possible that the new GC is buggy!
On the other hand, memory leaks of various kinds are bugs that should
be fixed when understood, and in the meantime features that trigger
them identified so you can avoid them.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-22 8:30 ` Stephen J. Turnbull
@ 2011-10-22 12:03 ` Carsten Mattner
2011-10-22 16:08 ` Stephen J. Turnbull
0 siblings, 1 reply; 113+ messages in thread
From: Carsten Mattner @ 2011-10-22 12:03 UTC (permalink / raw)
To: Stephen J. Turnbull; +Cc: Dmitry Antipov, Stefan Monnier, emacs-devel
On Sat, Oct 22, 2011 at 10:30 AM, Stephen J. Turnbull
<stephen@xemacs.org> wrote:
> Nobody is saying that running out of memory isn't a serious problem.
> Stefan's point is simply that "if you don't have enough RAM to use
> Emacs comfortably, you don't have enough RAM to use Emacs comfortably."
Sure.
Actually, if I could port the keybindings which work well with
X or Cocoa Emacs clients to the terminal emulator (-nw), I could
save even more memory.
Any tutorial/gotchas I can study to make my keybindings portable
between xterm, tmux, screen, Cocoa and X (no toolkit)?
The color discrepancy is a minor issue, which can be resolved
with selecting a theme that also looks correct in the term emulator.
> There are applications that use huge amounts of memory at
> initialization or in the early part of an algorithm, then can free it
> and never need it again. In those applications, freeing memory after
> use helps a lot. Emacs isn't one of those; average memory usage is
> normally a reasonably high fraction of peak usage (at least 75%, often
> effectively 100%, in my own usage patterns) and the peaks tend to
> recur. If that's more memory than you can afford, there's little we
> can do. Probably memory usage can be decreased on average or even at
> peak by a few percent, but is it worth the effort to develop and
> maintain very sophisticated allocators? While it is my belief that
> XEmacs's current excess memory consumption is related to system
> libraries like X, I have to admit that reports started to appear about
> the same time that a new garbage collection framework was introduced
> -- it's quite possible that the new GC is buggy!
A GC that doesn't fit the usage patterns may be worse than
no GC at all :). There's a reason we have many GC algos.
Erlang's copying GC fits the many-lightweight processes model
wel, for example. It's hard to use such a GC with the JVM's usual
workloads or GHC's, although GHC has and Erlang process like
the surrounding mechanics and different models included in GHC
make it harder. This is a case where elisp and Erlang have it easier,
I believe.
> On the other hand, memory leaks of various kinds are bugs that should
> be fixed when understood, and in the meantime features that trigger
> them identified so you can avoid them.
Is the gnutls memory usage a leak, or just some allocated arrays for
crypto arrays? Just a wild guess, nothing scientific to back it up.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-22 12:03 ` Carsten Mattner
@ 2011-10-22 16:08 ` Stephen J. Turnbull
2011-10-27 22:10 ` Ted Zlatanov
0 siblings, 1 reply; 113+ messages in thread
From: Stephen J. Turnbull @ 2011-10-22 16:08 UTC (permalink / raw)
To: Carsten Mattner; +Cc: Dmitry Antipov, Stefan Monnier, emacs-devel
Carsten Mattner writes:
> Is the gnutls memory usage a leak, or just some allocated arrays for
> crypto arrays? Just a wild guess, nothing scientific to back it up.
It sounds like a leak to me, since the reports we're seeing are of
memory creeping up over time. I would expect any arrays to be on the
order of a few MB, not 100s.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-22 16:08 ` Stephen J. Turnbull
@ 2011-10-27 22:10 ` Ted Zlatanov
2011-10-28 6:40 ` Eli Zaretskii
0 siblings, 1 reply; 113+ messages in thread
From: Ted Zlatanov @ 2011-10-27 22:10 UTC (permalink / raw)
To: emacs-devel
On Sun, 23 Oct 2011 01:08:15 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
SJT> Carsten Mattner writes:
>> Is the gnutls memory usage a leak, or just some allocated arrays for
>> crypto arrays? Just a wild guess, nothing scientific to back it up.
SJT> It sounds like a leak to me, since the reports we're seeing are of
SJT> memory creeping up over time. I would expect any arrays to be on the
SJT> order of a few MB, not 100s.
We have 3 reports from Sven Joachim, Nix, and Carsten Mattner. I will
do my best to find the issues. They seem to be in the C code I wrote to
link with GnuTLS.
It would be really helpful if someone who knows C debugging well could
run Valgrind or something similar against Emacs with GnuTLS compiled,
hitting all these leaks. I will take a look at the code, since I am
also reviewing it to prevent the lockups on hung connections, but
concrete Valgrind reports would be great.
Thanks
Ted
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-27 22:10 ` Ted Zlatanov
@ 2011-10-28 6:40 ` Eli Zaretskii
2011-10-28 9:34 ` Stephen J. Turnbull
0 siblings, 1 reply; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-28 6:40 UTC (permalink / raw)
To: emacs-devel; +Cc: emacs-devel
> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 27 Oct 2011 18:10:56 -0400
>
> It would be really helpful if someone who knows C debugging well could
> run Valgrind or something similar against Emacs with GnuTLS compiled,
> hitting all these leaks. I will take a look at the code, since I am
> also reviewing it to prevent the lockups on hung connections, but
> concrete Valgrind reports would be great.
Alas, I think Emacs will not run under valgrind.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-28 6:40 ` Eli Zaretskii
@ 2011-10-28 9:34 ` Stephen J. Turnbull
2011-10-28 9:47 ` Eli Zaretskii
0 siblings, 1 reply; 113+ messages in thread
From: Stephen J. Turnbull @ 2011-10-28 9:34 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
Eli Zaretskii writes:
> Alas, I think Emacs will not run under valgrind.
XEmacs does (at least some versions have done so ;-) so it's not
impossible in principle AFAIK. Maybe valgrind technology has improved
since you last tried, or maybe it only works on particular
architectures, but I'm pretty sure a couple of people were able to
valgrind XEmacs.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-28 9:34 ` Stephen J. Turnbull
@ 2011-10-28 9:47 ` Eli Zaretskii
2011-10-28 10:22 ` Stephen J. Turnbull
0 siblings, 1 reply; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-28 9:47 UTC (permalink / raw)
To: Stephen J. Turnbull; +Cc: emacs-devel
> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 28 Oct 2011 18:34:23 +0900
> Cc: emacs-devel@gnu.org
>
> Eli Zaretskii writes:
>
> > Alas, I think Emacs will not run under valgrind.
>
> XEmacs does (at least some versions have done so ;-) so it's not
> impossible in principle AFAIK.
Nothing is impossible in principle: this is software. It just could
be impractical or extremely hard.
One of the problems is dumping, and that of XEmacs uses an entirely
different method AFAIK. Maybe people should try running temacs
instead.
> Maybe valgrind technology has improved since you last tried, or
> maybe it only works on particular architectures, but I'm pretty sure
> a couple of people were able to valgrind XEmacs.
It definitely is worth trying again every now and then, yes.
If someone succeeds, please add the necessary know-how to etc/DEBUG.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-28 9:47 ` Eli Zaretskii
@ 2011-10-28 10:22 ` Stephen J. Turnbull
2011-10-28 10:36 ` Eli Zaretskii
0 siblings, 1 reply; 113+ messages in thread
From: Stephen J. Turnbull @ 2011-10-28 10:22 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
Eli Zaretskii writes:
> One of the problems is dumping, and that of XEmacs uses an entirely
> different method AFAIK.
True. I was just saying....
> Maybe people should try running temacs instead.
Ah, that rings a bell. Indeed, running from temacs may have been
necessary. I don't know about Emacs, but XEmacs has a recipe for
running a regular session from temacs in the Makefile (it's something
like "make run-temacs"). Emacs is probably different, but maybe you
already have the necessary magic spells for doing this? Or maybe it's
trivial for Emacs (it requires some special options for XEmacs, no big
deal but they're less than obvious or well-documented ;).
IIRC this takes a lot of extra time at startup, but once the dumpable
Lisp is loaded, temacs is otherwise no slower (but valgrind does cost
some performance).
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-28 10:22 ` Stephen J. Turnbull
@ 2011-10-28 10:36 ` Eli Zaretskii
2011-10-28 11:52 ` valgrind warnings [Re: Emacs bzr memory footprint] Dan Nicolaescu
0 siblings, 1 reply; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-28 10:36 UTC (permalink / raw)
To: Stephen J. Turnbull; +Cc: emacs-devel
> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 19:22:14 +0900
>
> > Maybe people should try running temacs instead.
>
> Ah, that rings a bell. Indeed, running from temacs may have been
> necessary. I don't know about Emacs, but XEmacs has a recipe for
> running a regular session from temacs in the Makefile (it's something
> like "make run-temacs"). Emacs is probably different, but maybe you
> already have the necessary magic spells for doing this? Or maybe it's
> trivial for Emacs (it requires some special options for XEmacs, no big
> deal but they're less than obvious or well-documented ;).
Dan made temacs runnable as you'd run any program. So I imagine
one needs to do this (100% untested):
./temacs -l loadup
M-x load-file RET ~/.emacs RET
(It is possible that the second line is not needed, but I guess anyone
should be able to see immediately if her customizations are or aren't
in effect.)
And thereafter use Emacs as usual.
Modifications of the above recipe to run under valgrind are left to
the interested parties ;-)
^ permalink raw reply [flat|nested] 113+ messages in thread
* valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 10:36 ` Eli Zaretskii
@ 2011-10-28 11:52 ` Dan Nicolaescu
2011-10-28 14:08 ` Stephen J. Turnbull
` (3 more replies)
0 siblings, 4 replies; 113+ messages in thread
From: Dan Nicolaescu @ 2011-10-28 11:52 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Stephen J. Turnbull, emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
> So I imagine one needs to do this (100% untested):
>
> ./temacs -l loadup
> M-x load-file RET ~/.emacs RET
Actually
./temacs
is enough.
On Fedora 15 with valgrind-3.6.1 it is possible to run
valgrind ./temacs
Valgrind produces these warnings, maybe some of them are interesting:
==30382== Conditional jump or move depends on uninitialised value(s)
==30382== at 0x5B7084: mark_object (alloc.c:5691)
==30382== by 0x5B6E71: mark_object (alloc.c:5612)
==30382== by 0x5B6DDF: mark_object (alloc.c:5591)
==30382== by 0x5B7109: mark_object (alloc.c:5702)
==30382== by 0x5B6E1D: mark_object (alloc.c:5595)
==30382== by 0x5B687B: mark_vectorlike (alloc.c:5391)
==30382== by 0x5B6DA8: mark_object (alloc.c:5579)
==30382== by 0x5B6DDF: mark_object (alloc.c:5591)
==30382== by 0x5B687B: mark_vectorlike (alloc.c:5391)
==30382== by 0x5B6DA8: mark_object (alloc.c:5579)
==30382== by 0x5B6DDF: mark_object (alloc.c:5591)
==30382== by 0x5B687B: mark_vectorlike (alloc.c:5391)
==30382==
==30382== Conditional jump or move depends on uninitialised value(s)
==30382== at 0x4731E6: note_mouse_highlight (xdisp.c:26841)
==30382== by 0x4F8682: note_mouse_movement (xterm.c:3830)
==30382== by 0x4FCB2E: handle_one_xevent (xterm.c:6652)
==30382== by 0x4FD792: XTread_socket (xterm.c:7133)
==30382== by 0x5437F7: read_avail_input (keyboard.c:6821)
==30382== by 0x54409F: handle_async_input (keyboard.c:7149)
==30382== by 0x5440BE: process_pending_signals (keyboard.c:7165)
==30382== by 0x62D713: wait_reading_process_output (process.c:4340)
==30382== by 0x419ED2: sit_for (dispnew.c:5971)
==30382== by 0x53AF87: read_char (keyboard.c:2687)
==30382== by 0x548876: read_key_sequence (keyboard.c:9290)
==30382== by 0x538709: command_loop_1 (keyboard.c:1447)
==30382== Conditional jump or move depends on uninitialised value(s)
==30382== at 0x46D785: x_write_glyphs (xdisp.c:24661)
==30382== by 0x413E4A: update_text_area (dispnew.c:3765)
==30382== by 0x414794: update_window_line (dispnew.c:4006)
==30382== by 0x413A1A: update_window (dispnew.c:3625)
==30382== by 0x413256: update_single_window (dispnew.c:3370)
==30382== by 0x511A93: Fx_show_tip (xfns.c:5189)
==30382== by 0x5D6E47: Ffuncall (eval.c:2995)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x6221C6: Fbyte_code (bytecode.c:423)
==30382== by 0x5D56AB: eval_sub (eval.c:2328)
==30382== by 0x5D381A: internal_lisp_condition_case (eval.c:1453)
==30382== by 0x62392A: exec_byte_code (bytecode.c:981)
==30382== Conditional jump or move depends on uninitialised value(s)
==30382== at 0x5B489D: live_misc_p (alloc.c:4001)
==30382== by 0x5B4C5D: mark_maybe_object (alloc.c:4138)
==30382== by 0x5B4FE6: mark_memory (alloc.c:4274)
==30382== by 0x5B507E: mark_stack (alloc.c:4532)
==30382== by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
==30382== by 0x5D69F0: Ffuncall (eval.c:2911)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382== Conditional jump or move depends on uninitialised value(s)
==30382== at 0x499251: coding_set_destination (coding.c:1046)
==30382== by 0x4B477A: decode_coding (coding.c:7022)
==30382== by 0x4B7317: decode_coding_object (coding.c:7671)
==30382== by 0x63610F: Fcall_process (callproc.c:813)
==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
==30382== by 0x5D5ECA: Fapply (eval.c:2479)
==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== Conditional jump or move depends on uninitialised value(s)
==30382== at 0x6302A9: send_process (process.c:5398)
==30382== by 0x630DC8: Fprocess_send_string (process.c:5648)
==30382== by 0x5D6D61: Ffuncall (eval.c:2977)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382== Conditional jump or move depends on uninitialised value(s)
==30382== at 0x636022: Fcall_process (callproc.c:799)
==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
==30382== by 0x5D5ABB: Fapply (eval.c:2422)
==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382== by 0x5D5ABB: Fapply (eval.c:2422)
==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
==30382== Use of uninitialised value of size 8
==30382== at 0x5B7000: mark_object (alloc.c:5674)
==30382== by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
==30382== by 0x5B4FE6: mark_memory (alloc.c:4274)
==30382== by 0x5B507E: mark_stack (alloc.c:4532)
==30382== by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
==30382== by 0x5D69F0: Ffuncall (eval.c:2911)
==30382== by 0x5D64CA: call2 (eval.c:2758)
==30382== by 0x5809AA: Finsert_file_contents (fileio.c:4144)
==30382== by 0x5D6E00: Ffuncall (eval.c:2990)
==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
^ permalink raw reply [flat|nested] 113+ messages in thread* valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 11:52 ` valgrind warnings [Re: Emacs bzr memory footprint] Dan Nicolaescu
@ 2011-10-28 14:08 ` Stephen J. Turnbull
2011-10-28 14:38 ` Dan Nicolaescu
` (2 subsequent siblings)
3 siblings, 0 replies; 113+ messages in thread
From: Stephen J. Turnbull @ 2011-10-28 14:08 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: Eli Zaretskii, emacs-devel
Dan Nicolaescu writes:
> Actually
> ./temacs
>
> is enough.
Nice work!
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 11:52 ` valgrind warnings [Re: Emacs bzr memory footprint] Dan Nicolaescu
2011-10-28 14:08 ` Stephen J. Turnbull
@ 2011-10-28 14:38 ` Dan Nicolaescu
2011-10-28 15:17 ` Chong Yidong
2011-10-28 15:27 ` Eli Zaretskii
2011-10-28 15:08 ` Ted Zlatanov
2011-10-28 20:20 ` Dan Nicolaescu
3 siblings, 2 replies; 113+ messages in thread
From: Dan Nicolaescu @ 2011-10-28 14:38 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Stephen J. Turnbull, emacs-devel
Dan Nicolaescu <dann@gnu.org> writes:
> valgrind ./temacs
>
> Valgrind produces these warnings, maybe some of them are interesting:
>
> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382== at 0x4731E6: note_mouse_highlight (xdisp.c:26841)
> ==30382== by 0x4F8682: note_mouse_movement (xterm.c:3830)
> ==30382== by 0x4FCB2E: handle_one_xevent (xterm.c:6652)
> ==30382== by 0x4FD792: XTread_socket (xterm.c:7133)
> ==30382== by 0x5437F7: read_avail_input (keyboard.c:6821)
> ==30382== by 0x54409F: handle_async_input (keyboard.c:7149)
> ==30382== by 0x5440BE: process_pending_signals (keyboard.c:7165)
> ==30382== by 0x62D713: wait_reading_process_output (process.c:4340)
> ==30382== by 0x419ED2: sit_for (dispnew.c:5971)
> ==30382== by 0x53AF87: read_char (keyboard.c:2687)
> ==30382== by 0x548876: read_key_sequence (keyboard.c:9290)
> ==30382== by 0x538709: command_loop_1 (keyboard.c:1447)
This is because window_from_coordinates in note_mouse_highlight is
passed "&part", where "part" is an uninitialized local variable.
When !NILP(window) "part" is not touched, so it stays uninitialized.
Can someone who understands this code fix this please?
> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382== at 0x46D785: x_write_glyphs (xdisp.c:24661)
> ==30382== by 0x413E4A: update_text_area (dispnew.c:3765)
> ==30382== by 0x414794: update_window_line (dispnew.c:4006)
> ==30382== by 0x413A1A: update_window (dispnew.c:3625)
> ==30382== by 0x413256: update_single_window (dispnew.c:3370)
> ==30382== by 0x511A93: Fx_show_tip (xfns.c:5189)
> ==30382== by 0x5D6E47: Ffuncall (eval.c:2995)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x6221C6: Fbyte_code (bytecode.c:423)
> ==30382== by 0x5D56AB: eval_sub (eval.c:2328)
> ==30382== by 0x5D381A: internal_lisp_condition_case (eval.c:1453)
> ==30382== by 0x62392A: exec_byte_code (bytecode.c:981)
I think I fixed this one.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 14:38 ` Dan Nicolaescu
@ 2011-10-28 15:17 ` Chong Yidong
2011-10-28 15:35 ` Eli Zaretskii
2011-10-28 15:27 ` Eli Zaretskii
1 sibling, 1 reply; 113+ messages in thread
From: Chong Yidong @ 2011-10-28 15:17 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: Eli Zaretskii, Stephen J. Turnbull, emacs-devel
Dan Nicolaescu <dann@gnu.org> writes:
> This is because window_from_coordinates in note_mouse_highlight is
> passed "&part", where "part" is an uninitialized local variable.
> When !NILP(window) "part" is not touched, so it stays uninitialized.
> Can someone who understands this code fix this please?
Fixed; thanks.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 15:17 ` Chong Yidong
@ 2011-10-28 15:35 ` Eli Zaretskii
0 siblings, 0 replies; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-28 15:35 UTC (permalink / raw)
To: Chong Yidong; +Cc: dann, stephen, emacs-devel
> From: Chong Yidong <cyd@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, "Stephen J. Turnbull" <stephen@xemacs.org>, emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 23:17:19 +0800
>
> Dan Nicolaescu <dann@gnu.org> writes:
>
> > This is because window_from_coordinates in note_mouse_highlight is
> > passed "&part", where "part" is an uninitialized local variable.
> > When !NILP(window) "part" is not touched, so it stays uninitialized.
> > Can someone who understands this code fix this please?
>
> Fixed; thanks.
I don't understand the fix.
--- src/xdisp.c 2011-10-19 11:46:17 +0000
+++ src/xdisp.c 2011-10-28 15:17:09 +0000
@@ -26835,11 +26835,14 @@ note_mouse_highlight (struct frame *f, i
/* Which window is that in? */
window = window_from_coordinates (f, x, y, &part, 1);
- /* If we were displaying active text in another window, clear that.
- Also clear if we move out of text area in same window. */
+ /* If displaying active text in another window, clear that. */
if (! EQ (window, hlinfo->mouse_face_window)
- || (part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
- && !NILP (hlinfo->mouse_face_window)))
+ /* Also clear if we move out of text area in same window. */
+ || (!NILP (hlinfo->mouse_face_window)
+ && !NILP (window)
+ && part != ON_TEXT
+ && part != ON_MODE_LINE
+ && part != ON_HEADER_LINE))
clear_mouse_face (hlinfo);
What is the purpose of testing that "window" is non-nil? How can it
be nil if it is EQ to hlinfo->mouse_face_window, which is non-nil?
That's the only real change in this revision, AFAICS.
What am I missing?
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 14:38 ` Dan Nicolaescu
2011-10-28 15:17 ` Chong Yidong
@ 2011-10-28 15:27 ` Eli Zaretskii
2011-10-28 16:34 ` Dan Nicolaescu
1 sibling, 1 reply; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-28 15:27 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: stephen, emacs-devel
> From: Dan Nicolaescu <dann@gnu.org>
> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>, emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 10:38:03 -0400
>
> > Valgrind produces these warnings, maybe some of them are interesting:
> >
> > ==30382== Conditional jump or move depends on uninitialised value(s)
> > ==30382== at 0x4731E6: note_mouse_highlight (xdisp.c:26841)
> > ==30382== by 0x4F8682: note_mouse_movement (xterm.c:3830)
> > ==30382== by 0x4FCB2E: handle_one_xevent (xterm.c:6652)
> > ==30382== by 0x4FD792: XTread_socket (xterm.c:7133)
> > ==30382== by 0x5437F7: read_avail_input (keyboard.c:6821)
> > ==30382== by 0x54409F: handle_async_input (keyboard.c:7149)
> > ==30382== by 0x5440BE: process_pending_signals (keyboard.c:7165)
> > ==30382== by 0x62D713: wait_reading_process_output (process.c:4340)
> > ==30382== by 0x419ED2: sit_for (dispnew.c:5971)
> > ==30382== by 0x53AF87: read_char (keyboard.c:2687)
> > ==30382== by 0x548876: read_key_sequence (keyboard.c:9290)
> > ==30382== by 0x538709: command_loop_1 (keyboard.c:1447)
>
> This is because window_from_coordinates in note_mouse_highlight is
> passed "&part", where "part" is an uninitialized local variable.
> When !NILP(window) "part" is not touched, so it stays uninitialized.
That's not my reading of the code. Here's what happens in
window_from_coordinates:
window = Qnil;
cw.window = &window, cw.x = x, cw.y = y; cw.part = part;
foreach_window (f, check_window_containing, &cw);
So cw.part now holds "part", which is a pointer.
check_window_containing does this:
found = coordinates_in_window (w, cw->x, cw->y);
if (found != ON_NOTHING)
{
*cw->part = found;
XSETWINDOW (*cw->window, w);
So if it finds a window, it touches "part" through cw->part and sets
`window' (through cw->window) to a non-nil value. Now back in
window_from_coordinates:
if (NILP (window)
&& tool_bar_p
&& WINDOWP (f->tool_bar_window)
&& WINDOW_TOTAL_LINES (XWINDOW (f->tool_bar_window)) > 0
&& (coordinates_in_window (XWINDOW (f->tool_bar_window), x, y)
!= ON_NOTHING))
{
*part = ON_TEXT;
window = f->tool_bar_window;
}
If `window' is non-nil, "part" was already set inside
check_window_containing, so there's no need to set it again.
Am I missing something or did this tricky code dupe valgrind?
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 15:27 ` Eli Zaretskii
@ 2011-10-28 16:34 ` Dan Nicolaescu
2011-10-28 18:36 ` Eli Zaretskii
0 siblings, 1 reply; 113+ messages in thread
From: Dan Nicolaescu @ 2011-10-28 16:34 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: stephen, emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
>> From: Dan Nicolaescu <dann@gnu.org>
>> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>, emacs-devel@gnu.org
>> Date: Fri, 28 Oct 2011 10:38:03 -0400
>>
>> > Valgrind produces these warnings, maybe some of them are interesting:
>> >
>> > ==30382== Conditional jump or move depends on uninitialised value(s)
>> > ==30382== at 0x4731E6: note_mouse_highlight (xdisp.c:26841)
>> > ==30382== by 0x4F8682: note_mouse_movement (xterm.c:3830)
>> > ==30382== by 0x4FCB2E: handle_one_xevent (xterm.c:6652)
>> > ==30382== by 0x4FD792: XTread_socket (xterm.c:7133)
>> > ==30382== by 0x5437F7: read_avail_input (keyboard.c:6821)
>> > ==30382== by 0x54409F: handle_async_input (keyboard.c:7149)
>> > ==30382== by 0x5440BE: process_pending_signals (keyboard.c:7165)
>> > ==30382== by 0x62D713: wait_reading_process_output (process.c:4340)
>> > ==30382== by 0x419ED2: sit_for (dispnew.c:5971)
>> > ==30382== by 0x53AF87: read_char (keyboard.c:2687)
>> > ==30382== by 0x548876: read_key_sequence (keyboard.c:9290)
>> > ==30382== by 0x538709: command_loop_1 (keyboard.c:1447)
>>
>> This is because window_from_coordinates in note_mouse_highlight is
>> passed "&part", where "part" is an uninitialized local variable.
>> When !NILP(window) "part" is not touched, so it stays uninitialized.
>
> That's not my reading of the code. Here's what happens in
> window_from_coordinates:
>
> window = Qnil;
> cw.window = &window, cw.x = x, cw.y = y; cw.part = part;
> foreach_window (f, check_window_containing, &cw);
>
> So cw.part now holds "part", which is a pointer.
> check_window_containing does this:
>
> found = coordinates_in_window (w, cw->x, cw->y);
> if (found != ON_NOTHING)
> {
> *cw->part = found;
> XSETWINDOW (*cw->window, w);
>
> So if it finds a window, it touches "part" through cw->part and sets
> `window' (through cw->window) to a non-nil value. Now back in
> window_from_coordinates:
>
> if (NILP (window)
> && tool_bar_p
> && WINDOWP (f->tool_bar_window)
> && WINDOW_TOTAL_LINES (XWINDOW (f->tool_bar_window)) > 0
> && (coordinates_in_window (XWINDOW (f->tool_bar_window), x, y)
> != ON_NOTHING))
> {
> *part = ON_TEXT;
> window = f->tool_bar_window;
> }
>
> If `window' is non-nil, "part" was already set inside
> check_window_containing, so there's no need to set it again.
>
> Am I missing something or did this tricky code dupe valgrind?
I don't know anything about this code...
A quick hack to initialize part with -42 and then abort if it's still -42 before
"part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE"
does indeed call abort...
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 16:34 ` Dan Nicolaescu
@ 2011-10-28 18:36 ` Eli Zaretskii
2011-10-28 18:51 ` Eli Zaretskii
2011-10-28 18:52 ` Dan Nicolaescu
0 siblings, 2 replies; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-28 18:36 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: stephen, emacs-devel
> From: Dan Nicolaescu <dann@gnu.org>
> Cc: stephen@xemacs.org, emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 12:34:07 -0400
>
> A quick hack to initialize part with -42 and then abort if it's still -42 before
> "part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE"
> does indeed call abort...
That's okay because window_from_coordinates is explicitly documented
to not touch "part" if the the position is not on any window. Can you
show me the exact code that you used for the above test?
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 18:36 ` Eli Zaretskii
@ 2011-10-28 18:51 ` Eli Zaretskii
2011-10-28 18:52 ` Dan Nicolaescu
1 sibling, 0 replies; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-28 18:51 UTC (permalink / raw)
To: dann, stephen, emacs-devel
> Date: Fri, 28 Oct 2011 20:36:58 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: stephen@xemacs.org, emacs-devel@gnu.org
>
> > From: Dan Nicolaescu <dann@gnu.org>
> > Cc: stephen@xemacs.org, emacs-devel@gnu.org
> > Date: Fri, 28 Oct 2011 12:34:07 -0400
> >
> > A quick hack to initialize part with -42 and then abort if it's still -42 before
> > "part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE"
> > does indeed call abort...
>
> That's okay because window_from_coordinates is explicitly documented
> to not touch "part" if the the position is not on any window. Can you
> show me the exact code that you used for the above test?
I fixed the problem by having "part" initialized inside
note_mouse_highlight to begin with.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 18:36 ` Eli Zaretskii
2011-10-28 18:51 ` Eli Zaretskii
@ 2011-10-28 18:52 ` Dan Nicolaescu
2011-10-28 19:03 ` Eli Zaretskii
1 sibling, 1 reply; 113+ messages in thread
From: Dan Nicolaescu @ 2011-10-28 18:52 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: stephen, emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
>> From: Dan Nicolaescu <dann@gnu.org>
>> Cc: stephen@xemacs.org, emacs-devel@gnu.org
>> Date: Fri, 28 Oct 2011 12:34:07 -0400
>>
>> A quick hack to initialize part with -42 and then abort if it's still -42 before
>> "part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE"
>> does indeed call abort...
>
> That's okay because window_from_coordinates is explicitly documented
> to not touch "part" if the the position is not on any window. Can you
> show me the exact code that you used for the above test?
=== modified file 'src/xdisp.c'
--- src/xdisp.c 2011-10-19 11:46:17 +0000
+++ src/xdisp.c 2011-10-28 16:29:15 +0000
@@ -26801,7 +26801,7 @@ void
note_mouse_highlight (struct frame *f, int x, int y)
{
Mouse_HLInfo *hlinfo = MOUSE_HL_INFO (f);
- enum window_part part;
+ enum window_part part = -42;
Lisp_Object window;
struct window *w;
Cursor cursor = No_Cursor;
@@ -26838,7 +26838,7 @@ note_mouse_highlight (struct frame *f, i
/* If we were displaying active text in another window, clear that.
Also clear if we move out of text area in same window. */
if (! EQ (window, hlinfo->mouse_face_window)
- || (part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
+ || ((part == -42 ? exit (1),1: 1) && part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
&& !NILP (hlinfo->mouse_face_window)))
clear_mouse_face (hlinfo);
This is before Yidong's fix.
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 18:52 ` Dan Nicolaescu
@ 2011-10-28 19:03 ` Eli Zaretskii
0 siblings, 0 replies; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-28 19:03 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: stephen, emacs-devel
> From: Dan Nicolaescu <dann@gnu.org>
> Cc: stephen@xemacs.org, emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 14:52:35 -0400
>
> === modified file 'src/xdisp.c'
> --- src/xdisp.c 2011-10-19 11:46:17 +0000
> +++ src/xdisp.c 2011-10-28 16:29:15 +0000
> @@ -26801,7 +26801,7 @@ void
> note_mouse_highlight (struct frame *f, int x, int y)
> {
> Mouse_HLInfo *hlinfo = MOUSE_HL_INFO (f);
> - enum window_part part;
> + enum window_part part = -42;
> Lisp_Object window;
> struct window *w;
> Cursor cursor = No_Cursor;
> @@ -26838,7 +26838,7 @@ note_mouse_highlight (struct frame *f, i
> /* If we were displaying active text in another window, clear that.
> Also clear if we move out of text area in same window. */
> if (! EQ (window, hlinfo->mouse_face_window)
> - || (part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
> + || ((part == -42 ? exit (1),1: 1) && part != ON_TEXT && part != ON_MODE_LINE && part != ON_HEADER_LINE
> && !NILP (hlinfo->mouse_face_window)))
> clear_mouse_face (hlinfo);
>
>
> This is before Yidong's fix.
Ah, now I understand. The call to `exit' should have been where
there's a call to clear_mouse_face, to see if the uninitialized value
really matters, because the `!NILP (hlinfo->mouse_face_window)' part
would have taken care of the -42 value (when `part' is not touched by
window_from_coordinates, `window' is nil, so you only get to test
`part' when hlinfo->mouse_face_window is also nil).
Anyway, the variable is now initialized.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 11:52 ` valgrind warnings [Re: Emacs bzr memory footprint] Dan Nicolaescu
2011-10-28 14:08 ` Stephen J. Turnbull
2011-10-28 14:38 ` Dan Nicolaescu
@ 2011-10-28 15:08 ` Ted Zlatanov
2011-12-04 21:11 ` Florian Weimer
2011-10-28 20:20 ` Dan Nicolaescu
3 siblings, 1 reply; 113+ messages in thread
From: Ted Zlatanov @ 2011-10-28 15:08 UTC (permalink / raw)
To: emacs-devel
On Fri, 28 Oct 2011 07:52:12 -0400 Dan Nicolaescu <dann@gnu.org> wrote:
DN> On Fedora 15 with valgrind-3.6.1 it is possible to run
DN> valgrind ./temacs
I ran this and opened and closed a GnuTLS connection many times:
(open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
I did not see any errors printed when I did this, but on exit a lot of
information was printed that did not seem related (I could not analyze
it all, it was too much data and I don't know what to look for other
than "gnutls"). Is this enough to tell us there are no memory leaks in
the "open connection" part of the code? I'll test sending and receiving
data if so.
Ted
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 15:08 ` Ted Zlatanov
@ 2011-12-04 21:11 ` Florian Weimer
0 siblings, 0 replies; 113+ messages in thread
From: Florian Weimer @ 2011-12-04 21:11 UTC (permalink / raw)
To: emacs-devel
* Ted Zlatanov:
> On Fri, 28 Oct 2011 07:52:12 -0400 Dan Nicolaescu <dann@gnu.org> wrote:
> DN> On Fedora 15 with valgrind-3.6.1 it is possible to run
>
> DN> valgrind ./temacs
>
> I ran this and opened and closed a GnuTLS connection many times:
>
> (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
>
> I did not see any errors printed when I did this, but on exit a lot of
> information was printed that did not seem related (I could not analyze
> it all, it was too much data and I don't know what to look for other
> than "gnutls"). Is this enough to tell us there are no memory leaks in
> the "open connection" part of the code? I'll test sending and receiving
> data if so.
I think you need to run
valgrind --leak-check=yes ./temacs
and actually terminate the program to get the dump of leaked
resources.
Has the bug been identified in the meantime? I've got some very
long-running processes using GnuTLS which grow without bounds,
too---but I've been blaming this on bugs in my Perl wrapper. In that
cases, just opening and closing connections (and sending a bit of
data) does not trigger the leak condition, either.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 11:52 ` valgrind warnings [Re: Emacs bzr memory footprint] Dan Nicolaescu
` (2 preceding siblings ...)
2011-10-28 15:08 ` Ted Zlatanov
@ 2011-10-28 20:20 ` Dan Nicolaescu
2011-10-29 3:40 ` Chong Yidong
2011-10-29 9:10 ` Eli Zaretskii
3 siblings, 2 replies; 113+ messages in thread
From: Dan Nicolaescu @ 2011-10-28 20:20 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
Dan Nicolaescu <dann@gnu.org> writes:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>> So I imagine one needs to do this (100% untested):
>>
>> ./temacs -l loadup
>> M-x load-file RET ~/.emacs RET
>
> Actually
> ./temacs
>
> is enough.
>
> On Fedora 15 with valgrind-3.6.1 it is possible to run
>
> valgrind ./temacs
>
> Valgrind produces these warnings, maybe some of them are interesting:
>
> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382== at 0x499251: coding_set_destination (coding.c:1046)
> ==30382== by 0x4B477A: decode_coding (coding.c:7022)
> ==30382== by 0x4B7317: decode_coding_object (coding.c:7671)
> ==30382== by 0x63610F: Fcall_process (callproc.c:813)
> ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382== by 0x5D5ECA: Fapply (eval.c:2479)
> ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
This looks like setup_coding_system (or whatever else initializes struct
coding_system) does not initialize the src_pos field.
>
> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382== at 0x6302A9: send_process (process.c:5398)
> ==30382== by 0x630DC8: Fprocess_send_string (process.c:5648)
> ==30382== by 0x5D6D61: Ffuncall (eval.c:2977)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
similarly the struct coding_system.src_multibyte needs to be
initialized.
> ==30382== Conditional jump or move depends on uninitialised value(s)
> ==30382== at 0x636022: Fcall_process (callproc.c:799)
> ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382== by 0x5D5ABB: Fapply (eval.c:2422)
> ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
> ==30382== by 0x5D5ABB: Fapply (eval.c:2422)
> ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
similarly the struct coding_system.dst_multibyte needs to be
initialized.
> ==30382== Use of uninitialised value of size 8
> ==30382== at 0x5B7000: mark_object (alloc.c:5674)
> ==30382== by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
> ==30382== by 0x5B4FE6: mark_memory (alloc.c:4274)
> ==30382== by 0x5B507E: mark_stack (alloc.c:4532)
> ==30382== by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
> ==30382== by 0x5D69F0: Ffuncall (eval.c:2911)
> ==30382== by 0x5D64CA: call2 (eval.c:2758)
> ==30382== by 0x5809AA: Finsert_file_contents (fileio.c:4144)
> ==30382== by 0x5D6E00: Ffuncall (eval.c:2990)
> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
It would be interesting to know if the gc warnings are relevant...
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 20:20 ` Dan Nicolaescu
@ 2011-10-29 3:40 ` Chong Yidong
2011-10-29 7:04 ` Andreas Schwab
2011-10-29 9:10 ` Eli Zaretskii
1 sibling, 1 reply; 113+ messages in thread
From: Chong Yidong @ 2011-10-29 3:40 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: Eli Zaretskii, emacs-devel
Dan Nicolaescu <dann@gnu.org> writes:
>> ==30382== Use of uninitialised value of size 8
>> ==30382== at 0x5B7000: mark_object (alloc.c:5674)
>> ==30382== by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
>> ==30382== by 0x5B4FE6: mark_memory (alloc.c:4274)
>> ==30382== by 0x5B507E: mark_stack (alloc.c:4532)
>> ==30382== by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
>> ==30382== by 0x5D69F0: Ffuncall (eval.c:2911)
>> ==30382== by 0x5D64CA: call2 (eval.c:2758)
>> ==30382== by 0x5809AA: Finsert_file_contents (fileio.c:4144)
>> ==30382== by 0x5D6E00: Ffuncall (eval.c:2990)
>> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
>> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
>> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
>
> It would be interesting to know if the gc warnings are relevant...
This is complaining about the plist member of struct Lisp_Overlay, but I
don't see how an uninitalized value could creep in there. Overlays are
only allocated in copy_overlays and Fmake_overlay, and in both places
the plist member appears to be assigned correctly.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-29 3:40 ` Chong Yidong
@ 2011-10-29 7:04 ` Andreas Schwab
2011-10-29 16:25 ` Stefan Monnier
0 siblings, 1 reply; 113+ messages in thread
From: Andreas Schwab @ 2011-10-29 7:04 UTC (permalink / raw)
To: Chong Yidong; +Cc: Dan Nicolaescu, Eli Zaretskii, emacs-devel
Chong Yidong <cyd@gnu.org> writes:
> Dan Nicolaescu <dann@gnu.org> writes:
>
>>> ==30382== Use of uninitialised value of size 8
>>> ==30382== at 0x5B7000: mark_object (alloc.c:5674)
>>> ==30382== by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
>>> ==30382== by 0x5B4FE6: mark_memory (alloc.c:4274)
>>> ==30382== by 0x5B507E: mark_stack (alloc.c:4532)
>>> ==30382== by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
>>> ==30382== by 0x5D69F0: Ffuncall (eval.c:2911)
>>> ==30382== by 0x5D64CA: call2 (eval.c:2758)
>>> ==30382== by 0x5809AA: Finsert_file_contents (fileio.c:4144)
>>> ==30382== by 0x5D6E00: Ffuncall (eval.c:2990)
>>> ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
>>> ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
>>> ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
>>
>> It would be interesting to know if the gc warnings are relevant...
>
> This is complaining about the plist member of struct Lisp_Overlay, but I
> don't see how an uninitalized value could creep in there. Overlays are
> only allocated in copy_overlays and Fmake_overlay, and in both places
> the plist member appears to be assigned correctly.
This is while scanning the stack, where it is apparently possible that
this happens to look like a Lisp_Object, but isn't.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-29 7:04 ` Andreas Schwab
@ 2011-10-29 16:25 ` Stefan Monnier
2011-11-04 12:51 ` Dan Nicolaescu
0 siblings, 1 reply; 113+ messages in thread
From: Stefan Monnier @ 2011-10-29 16:25 UTC (permalink / raw)
To: Andreas Schwab; +Cc: Dan Nicolaescu, Eli Zaretskii, Chong Yidong, emacs-devel
> This is while scanning the stack, where it is apparently possible that
> this happens to look like a Lisp_Object, but isn't.
That doesn't explain it, tho: the conservative stack scanning does check
to make sure the suspected pointer really does point to a real (tho
possibly dead) object, and we always know its type.
I.e. barring bugs in the conservative-scanning code, it should only
cause dead objects to be kept around (and scanned). But those used to
be live, so they have been initialized.
Stefan
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-29 16:25 ` Stefan Monnier
@ 2011-11-04 12:51 ` Dan Nicolaescu
2011-11-04 13:08 ` Andreas Schwab
` (2 more replies)
0 siblings, 3 replies; 113+ messages in thread
From: Dan Nicolaescu @ 2011-11-04 12:51 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Eli Zaretskii, Chong Yidong, Andreas Schwab, emacs-devel
Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> This is while scanning the stack, where it is apparently possible that
>> this happens to look like a Lisp_Object, but isn't.
>
> That doesn't explain it, tho: the conservative stack scanning does check
> to make sure the suspected pointer really does point to a real (tho
> possibly dead) object, and we always know its type.
It turns out there's a --track-origins=yes valgrind flag. With that
turned on we get this warning:
==9622== Conditional jump or move depends on uninitialised value(s)
==9622== at 0x5B3D89: mem_find (alloc.c:3521)
==9622== by 0x5B4AEB: mark_maybe_object (alloc.c:4103)
==9622== by 0x5B50E6: mark_memory (alloc.c:4275)
==9622== by 0x5B517E: mark_stack (alloc.c:4533)
==9622== by 0x5B6048: Fgarbage_collect (alloc.c:5120)
==9622== by 0x5D542C: eval_sub (eval.c:2217)
==9622== by 0x602A0B: readevalloop (lread.c:1838)
==9622== by 0x6015B6: Fload (lread.c:1316)
==9622== by 0x5D59FF: eval_sub (eval.c:2336)
==9622== by 0x602A0B: readevalloop (lread.c:1838)
==9622== by 0x6015B6: Fload (lread.c:1316)
==9622== by 0x5D59FF: eval_sub (eval.c:2336)
==9622== Uninitialised value was created by a stack allocation
==9622== at 0x5B5D00: Fgarbage_collect (alloc.c:4993)
==9622==
And indeed, there's:
Lisp_Object total[8];
in Fgarbage_collect.
What should we do about this?
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-11-04 12:51 ` Dan Nicolaescu
@ 2011-11-04 13:08 ` Andreas Schwab
2011-11-04 15:50 ` valgrind warnings Paul Eggert
2011-11-04 17:37 ` valgrind warnings [Re: Emacs bzr memory footprint] Stefan Monnier
2 siblings, 0 replies; 113+ messages in thread
From: Andreas Schwab @ 2011-11-04 13:08 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: Eli Zaretskii, Chong Yidong, Stefan Monnier, emacs-devel
Dan Nicolaescu <dann@gnu.org> writes:
> What should we do about this?
It is obvious that the stack scanning can find all kinds of stale data,
but that is inherent to conservative gc.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings
2011-11-04 12:51 ` Dan Nicolaescu
2011-11-04 13:08 ` Andreas Schwab
@ 2011-11-04 15:50 ` Paul Eggert
2011-11-04 17:37 ` valgrind warnings [Re: Emacs bzr memory footprint] Stefan Monnier
2 siblings, 0 replies; 113+ messages in thread
From: Paul Eggert @ 2011-11-04 15:50 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: emacs-devel
On 11/04/11 05:51, Dan Nicolaescu wrote:
> What should we do about this?
There's nothing wrong with the code; this is a false
alarm of valgrind, which is inherent to the way a
conservative garbage collector works.
One way to work around this valgrind issue is to supply
a suppression file, e.g., put something like the
following into a new file emacs/src/valgrind.supp:
{
Conservative garbage collection Cond
Memcheck:Cond
...
fun:mark_maybe_*
}
{
Conservative garbage collection Value8
Memcheck:Value8
...
fun:mark_maybe_*
}
(this is just an indication of what it would look
like; more is needed). The goal is to suppress
false positives while minimizing false negatives.
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-11-04 12:51 ` Dan Nicolaescu
2011-11-04 13:08 ` Andreas Schwab
2011-11-04 15:50 ` valgrind warnings Paul Eggert
@ 2011-11-04 17:37 ` Stefan Monnier
2011-11-04 19:10 ` Dan Nicolaescu
2 siblings, 1 reply; 113+ messages in thread
From: Stefan Monnier @ 2011-11-04 17:37 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: Eli Zaretskii, Chong Yidong, Andreas Schwab, emacs-devel
>>> This is while scanning the stack, where it is apparently possible that
>>> this happens to look like a Lisp_Object, but isn't.
>> That doesn't explain it, tho: the conservative stack scanning does check
>> to make sure the suspected pointer really does point to a real (tho
>> possibly dead) object, and we always know its type.
> It turns out there's a --track-origins=yes valgrind flag. With that
> turned on we get this warning:
> ==9622== Conditional jump or move depends on uninitialised value(s)
> ==9622== at 0x5B3D89: mem_find (alloc.c:3521)
> ==9622== by 0x5B4AEB: mark_maybe_object (alloc.c:4103)
> ==9622== by 0x5B50E6: mark_memory (alloc.c:4275)
> ==9622== by 0x5B517E: mark_stack (alloc.c:4533)
> ==9622== by 0x5B6048: Fgarbage_collect (alloc.c:5120)
> ==9622== by 0x5D542C: eval_sub (eval.c:2217)
> ==9622== by 0x602A0B: readevalloop (lread.c:1838)
> ==9622== by 0x6015B6: Fload (lread.c:1316)
> ==9622== by 0x5D59FF: eval_sub (eval.c:2336)
> ==9622== by 0x602A0B: readevalloop (lread.c:1838)
> ==9622== by 0x6015B6: Fload (lread.c:1316)
> ==9622== by 0x5D59FF: eval_sub (eval.c:2336)
> ==9622== Uninitialised value was created by a stack allocation
> ==9622== at 0x5B5D00: Fgarbage_collect (alloc.c:4993)
> ==9622==
> And indeed, there's:
> Lisp_Object total[8];
> in Fgarbage_collect.
Ah, right, so the problem is not that the stack "points" to some
uninitialized value, but that the stack itself contains uninitialized
values. Yes, that's perfectly harmless (at least no worse than the
arbitrary initialized values we can find).
> What should we do about this?
Tell valgrind that it's a non-issue.
Stefan
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-11-04 17:37 ` valgrind warnings [Re: Emacs bzr memory footprint] Stefan Monnier
@ 2011-11-04 19:10 ` Dan Nicolaescu
2011-11-04 20:56 ` Paul Eggert
0 siblings, 1 reply; 113+ messages in thread
From: Dan Nicolaescu @ 2011-11-04 19:10 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Eli Zaretskii, Chong Yidong, Andreas Schwab, emacs-devel
[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]
Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> This is while scanning the stack, where it is apparently possible that
>>>> this happens to look like a Lisp_Object, but isn't.
>>> That doesn't explain it, tho: the conservative stack scanning does check
>>> to make sure the suspected pointer really does point to a real (tho
>>> possibly dead) object, and we always know its type.
>> It turns out there's a --track-origins=yes valgrind flag. With that
>> turned on we get this warning:
>
>> ==9622== Conditional jump or move depends on uninitialised value(s)
>> ==9622== at 0x5B3D89: mem_find (alloc.c:3521)
>> ==9622== by 0x5B4AEB: mark_maybe_object (alloc.c:4103)
>> ==9622== by 0x5B50E6: mark_memory (alloc.c:4275)
>> ==9622== by 0x5B517E: mark_stack (alloc.c:4533)
>> ==9622== by 0x5B6048: Fgarbage_collect (alloc.c:5120)
>> ==9622== by 0x5D542C: eval_sub (eval.c:2217)
>> ==9622== by 0x602A0B: readevalloop (lread.c:1838)
>> ==9622== by 0x6015B6: Fload (lread.c:1316)
>> ==9622== by 0x5D59FF: eval_sub (eval.c:2336)
>> ==9622== by 0x602A0B: readevalloop (lread.c:1838)
>> ==9622== by 0x6015B6: Fload (lread.c:1316)
>> ==9622== by 0x5D59FF: eval_sub (eval.c:2336)
>> ==9622== Uninitialised value was created by a stack allocation
>> ==9622== at 0x5B5D00: Fgarbage_collect (alloc.c:4993)
>> ==9622==
>
>> And indeed, there's:
>> Lisp_Object total[8];
>> in Fgarbage_collect.
>
> Ah, right, so the problem is not that the stack "points" to some
> uninitialized value, but that the stack itself contains uninitialized
> values. Yes, that's perfectly harmless (at least no worse than the
> arbitrary initialized values we can find).
>
>> What should we do about this?
>
> Tell valgrind that it's a non-issue.
There is a way to do that, write a configuration file and pass it to
valgrind with: --suppressions=FILENAME
Such a file is attached.
I think it should be checked in src/ so that everyone can make use of it
and improve it. OK to check in?
[-- Attachment #2: valgrind.supp --]
[-- Type: text/plain, Size: 1242 bytes --]
# valgrind supression file
# generated on a x86_64-unknown-linux-gnu machine
# Usage:
# valgrind --suppressions=valgrind.supp ./temacs
# This looks like this:
# Invalid read of size 4
# at 0x511B3C: init_buffer (buffer.c:5037)
# by 0x40E293: main (emacs.c:1452)
# Address 0x50cff24 is 68 bytes inside a block of size 70 alloc'd
# at 0x4A06582: realloc (vg_replace_malloc.c:525)
# by 0x511B23: init_buffer (buffer.c:5071)
# by 0x40E293: main (emacs.c:1452)
#
# It is due to the optimized strlen implementation that loads 32 bits at a time.
{
optimized_strlen_in_init_buffer
Memcheck:Addr4
fun:init_buffer
fun:main
}
# Disable various warnings due to garbage collection scanning the stack.
{
<GCcond>
Memcheck:Cond
fun:Fgarbage_collect
fun:*
}
{
<GCvalue>
Memcheck:Value8
fun:Fgarbage_collect
fun:*
}
{
<mark>
Memcheck:Value8
fun:mark_object
fun:*
}
{
mark_object_cond
Memcheck:Cond
fun:mark_object
fun:*
}
{
mark_vectorlike_value8
Memcheck:Value8
fun:mark_vectorlike
fun:Fgarbage_collect
fun:*
}
{
survives_gc_p_cond
Memcheck:Cond
fun:survives_gc_p
fun:sweep_weak_table
fun:sweep_weak_hash_tables
fun:Fgarbage_collect
fun:*
}
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-11-04 19:10 ` Dan Nicolaescu
@ 2011-11-04 20:56 ` Paul Eggert
2011-11-05 22:08 ` Dan Nicolaescu
0 siblings, 1 reply; 113+ messages in thread
From: Paul Eggert @ 2011-11-04 20:56 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 551 bytes --]
I like the idea of creating a src/valgrind.supp file.
I looked at Dan's proposal and merged it with mine,
adding some comments to try to explain things
(merged version attached).
One question/problem that I had with Dan's is this rule:
{
survives_gc_p_cond
Memcheck:Cond
fun:survives_gc_p
fun:sweep_weak_table
fun:sweep_weak_hash_tables
fun:Fgarbage_collect
fun:*
}
As near as I can make out, if survives_gc_p is accessing
uninitialized storage, it's buggy, no? Shouldn't we omit
this rule and fix the bug instead?
[-- Attachment #2: valgrind.supp --]
[-- Type: text/plain, Size: 1408 bytes --]
# valgrind suppression file
# Usage:
# valgrind --suppressions=valgrind.supp ./temacs
# Conservative garbage collection inherently looks at uninitialized values,
# and mark_maybe_object is at the core of this.
{
mark_maybe_object Cond - Conservative garbage collection
Memcheck:Cond
fun:mark_maybe_object
}
{
mark_maybe_object Value8 - Conservative garbage collection
Memcheck:Value8
fun:mark_maybe_object
}
# mark_maybe_object can be inlined from Fgarbage_collect.
{
mark_maybe_object Cond - Conservative garbage collection
Memcheck:Cond
fun:Fgarbage_collect
}
{
mark_maybe_object Value8 - Conservative garbage collection
Memcheck:Value8
fun:Fgarbage_collect
}
# mark_maybe_object can be inlined from mark_object.
{
mark_maybe_object Cond - Conservative garbage collection
Memcheck:Cond
fun:mark_object
}
{
mark_maybe_object Value8 - Conservative garbage collection
Memcheck:Value8
fun:mark_object
}
# mark_maybe_object can be inlined from mark_vectorlike.
{
mark_vectorlike Cond - Conservative garbage collection
Memcheck:Cond
fun:mark_vectorlike
}
{
mark_vectorlike Value8 - Conservative garbage collection
Memcheck:Value8
fun:mark_vectorlike
}
# On one circa-2011 x86-64 GNU/Linux platform, strlen is inlined to
# something that loads 4 bytes at a time.
{
init_buffer optimized strlen
Memcheck:Addr4
fun:init_buffer
}
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-11-04 20:56 ` Paul Eggert
@ 2011-11-05 22:08 ` Dan Nicolaescu
2011-11-06 2:34 ` Paul Eggert
0 siblings, 1 reply; 113+ messages in thread
From: Dan Nicolaescu @ 2011-11-05 22:08 UTC (permalink / raw)
To: Paul Eggert; +Cc: emacs-devel
Paul Eggert <eggert@cs.ucla.edu> writes:
> I like the idea of creating a src/valgrind.supp file.
> I looked at Dan's proposal and merged it with mine,
> adding some comments to try to explain things
> (merged version attached).
This version looks nicer!
> One question/problem that I had with Dan's is this rule:
>
> {
> survives_gc_p_cond
> Memcheck:Cond
> fun:survives_gc_p
> fun:sweep_weak_table
> fun:sweep_weak_hash_tables
> fun:Fgarbage_collect
> fun:*
> }
>
> As near as I can make out, if survives_gc_p is accessing
> uninitialized storage, it's buggy, no? Shouldn't we omit
> this rule and fix the bug instead?
The full warning looks like this (when also using --track-origins=yes):
==4937== Conditional jump or move depends on uninitialised value(s)
==4937== at 0x5B7567: survives_gc_p (alloc.c:5838)
==4937== by 0x5E5FBD: sweep_weak_table (fns.c:3965)
==4937== by 0x5E62DE: sweep_weak_hash_tables (fns.c:4066)
==4937== by 0x5B75C8: gc_sweep (alloc.c:5850)
==4937== by 0x5B6195: Fgarbage_collect (alloc.c:5203)
==4937== by 0x62301A: exec_byte_code (bytecode.c:820)
==4937== by 0x5D7A96: funcall_lambda (eval.c:3205)
==4937== by 0x5D754A: apply_lambda (eval.c:3082)
==4937== by 0x5D5B4C: eval_sub (eval.c:2367)
==4937== by 0x5D3AF6: internal_lisp_condition_case (eval.c:1453)
==4937== by 0x623C06: exec_byte_code (bytecode.c:981)
==4937== by 0x5D7A96: funcall_lambda (eval.c:3205)
==4937== Uninitialised value was created by a stack allocation
==4937== at 0x6224AA: exec_byte_code (bytecode.c:437)
==4937==
Do you think that's a problem?
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-11-05 22:08 ` Dan Nicolaescu
@ 2011-11-06 2:34 ` Paul Eggert
2011-11-06 12:37 ` Dan Nicolaescu
0 siblings, 1 reply; 113+ messages in thread
From: Paul Eggert @ 2011-11-06 2:34 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: emacs-devel
On 11/05/11 15:08, Dan Nicolaescu wrote:
> Do you think that's a problem?
It looks suspicious, yes.
I don't have that problem with bzr 106302
(Fedora 15 x86-64, Emacs compiled with GCC 4.6.2).
I do recall seeing it in earlier bzrs on the trunk, though.
What happens when you run:
valgrind --db-attach=yes --suppressions=valgrind.supp ./temacs
Can you use GDB to navigate to the offending uninitialized value
and see why the garbage collector is trying to look
at that stack location?
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-11-06 2:34 ` Paul Eggert
@ 2011-11-06 12:37 ` Dan Nicolaescu
2011-11-07 1:13 ` Paul Eggert
0 siblings, 1 reply; 113+ messages in thread
From: Dan Nicolaescu @ 2011-11-06 12:37 UTC (permalink / raw)
To: Paul Eggert; +Cc: emacs-devel
Paul Eggert <eggert@cs.ucla.edu> writes:
> On 11/05/11 15:08, Dan Nicolaescu wrote:
>> Do you think that's a problem?
>
> It looks suspicious, yes.
>
> I don't have that problem with bzr 106302
> (Fedora 15 x86-64, Emacs compiled with GCC 4.6.2).
> I do recall seeing it in earlier bzrs on the trunk, though.
I use the same platform, but the compiler is
gcc (GCC) 4.6.1 20110908 (Red Hat 4.6.1-9)
I compile with -g -O0.
> What happens when you run:
>
> valgrind --db-attach=yes --suppressions=valgrind.supp ./temacs
>
> Can you use GDB to navigate to the offending uninitialized value
> and see why the garbage collector is trying to look
> at that stack location?
Tried that, but I didn't find anything that made sense...
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-11-06 12:37 ` Dan Nicolaescu
@ 2011-11-07 1:13 ` Paul Eggert
0 siblings, 0 replies; 113+ messages in thread
From: Paul Eggert @ 2011-11-07 1:13 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 267 bytes --]
On 11/06/11 04:37, Dan Nicolaescu wrote:
> Tried that, but I didn't find anything that made sense...
Valgrind isn't always supposed to make sense :-)
How about the attached valgrind.supp instead?
It suppresses more diagnostics, and with luck it'll suppress yours.
[-- Attachment #2: valgrind.supp --]
[-- Type: text/plain, Size: 1337 bytes --]
# valgrind suppression file
# Usage:
# valgrind --suppressions=valgrind.supp ./temacs
# Conservative garbage collection inherently looks at uninitialized values,
# and Fgarbage_collect and its callees all depend on this.
# It's hard to separate out exactly which callees need to be listed here,
# since the C compiler can inline them. Also, valgrind doesn't care
# about the use of uninitialized variables directly, only when their values
# are eventually used. So just list Fgarbage_collect and its callees.
{
Fgarbage_collect Cond - conservative garbage collection
Memcheck:Cond
...
fun:Fgarbage_collect
}
{
Fgarbage_collect Value8 - conservative garbage collection
Memcheck:Value8
...
fun:Fgarbage_collect
}
# valgrind only looks at the last few callees on the stack, but
# mark_object can call itself recursively and deeply. So list
# it too, in case Fgarbage_collect is a long way from the stack top.
{
Fgarbage_collect Cond - conservative garbage collection
Memcheck:Cond
...
fun:mark_object
}
{
Fgarbage_collect Value8 - conservative garbage collection
Memcheck:Value8
...
fun:mark_object
}
# On one circa-2011 x86-64 GNU/Linux platform, strlen is inlined to
# something that loads 4 bytes at a time.
{
init_buffer optimized strlen
Memcheck:Addr4
fun:init_buffer
}
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: valgrind warnings [Re: Emacs bzr memory footprint]
2011-10-28 20:20 ` Dan Nicolaescu
2011-10-29 3:40 ` Chong Yidong
@ 2011-10-29 9:10 ` Eli Zaretskii
1 sibling, 0 replies; 113+ messages in thread
From: Eli Zaretskii @ 2011-10-29 9:10 UTC (permalink / raw)
To: Dan Nicolaescu; +Cc: emacs-devel
> From: Dan Nicolaescu <dann@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 28 Oct 2011 16:20:25 -0400
>
> > Valgrind produces these warnings, maybe some of them are interesting:
> >
> > ==30382== Conditional jump or move depends on uninitialised value(s)
> > ==30382== at 0x499251: coding_set_destination (coding.c:1046)
> > ==30382== by 0x4B477A: decode_coding (coding.c:7022)
> > ==30382== by 0x4B7317: decode_coding_object (coding.c:7671)
> > ==30382== by 0x63610F: Fcall_process (callproc.c:813)
> > ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382== by 0x5D5ECA: Fapply (eval.c:2479)
> > ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
> > ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
>
> This looks like setup_coding_system (or whatever else initializes struct
> coding_system) does not initialize the src_pos field.
>
> >
> > ==30382== Conditional jump or move depends on uninitialised value(s)
> > ==30382== at 0x6302A9: send_process (process.c:5398)
> > ==30382== by 0x630DC8: Fprocess_send_string (process.c:5648)
> > ==30382== by 0x5D6D61: Ffuncall (eval.c:2977)
> > ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
> > ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
> > ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
>
> similarly the struct coding_system.src_multibyte needs to be
> initialized.
>
>
> > ==30382== Conditional jump or move depends on uninitialised value(s)
> > ==30382== at 0x636022: Fcall_process (callproc.c:799)
> > ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382== by 0x5D5ABB: Fapply (eval.c:2422)
> > ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
> > ==30382== by 0x5D5ABB: Fapply (eval.c:2422)
> > ==30382== by 0x5D6C07: Ffuncall (eval.c:2956)
> > ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
>
> similarly the struct coding_system.dst_multibyte needs to be
> initialized.
>
>
> > ==30382== Use of uninitialised value of size 8
> > ==30382== at 0x5B7000: mark_object (alloc.c:5674)
> > ==30382== by 0x5B4C9B: mark_maybe_object (alloc.c:4152)
> > ==30382== by 0x5B4FE6: mark_memory (alloc.c:4274)
> > ==30382== by 0x5B507E: mark_stack (alloc.c:4532)
> > ==30382== by 0x5B5F48: Fgarbage_collect (alloc.c:5119)
> > ==30382== by 0x5D69F0: Ffuncall (eval.c:2911)
> > ==30382== by 0x5D64CA: call2 (eval.c:2758)
> > ==30382== by 0x5809AA: Finsert_file_contents (fileio.c:4144)
> > ==30382== by 0x5D6E00: Ffuncall (eval.c:2990)
> > ==30382== by 0x622C5B: exec_byte_code (bytecode.c:785)
> > ==30382== by 0x5D77BA: funcall_lambda (eval.c:3205)
> > ==30382== by 0x5D6F4C: Ffuncall (eval.c:3023)
>
> It would be interesting to know if the gc warnings are relevant...
I suggest to file a separate bug report about each valgrind warning.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-22 7:34 ` Carsten Mattner
2011-10-22 8:30 ` Stephen J. Turnbull
@ 2011-10-22 8:44 ` Sven Joachim
2011-10-22 11:57 ` Carsten Mattner
2011-10-23 18:04 ` Stefan Monnier
2 siblings, 1 reply; 113+ messages in thread
From: Sven Joachim @ 2011-10-22 8:44 UTC (permalink / raw)
To: Carsten Mattner; +Cc: Dmitry Antipov, Stefan Monnier, emacs-devel
On 2011-10-22 09:34 +0200, Carsten Mattner wrote:
> On Fri, Oct 21, 2011 at 8:00 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>> > This creates two-level fragmentation: the whole heap is fragmented as seen
>> > by malloc(), and allocated space is fragmented too, as seen by Emacs. Since
>>
>> Yes.
>>
>> > solid workload will consume 100M of the heap, the most of this 90M will be re-used -
>>
>> Exactly: so this fragmentation is not a serious problem in the sense
>> that this free memory can be (re)used by Emacs.
>>
>> > but it will never freed.
>>
>> Right, so it's a problem in the sense that this memory will not be
>> available to other processes.
>>
>> While this is not perfect, I don't consider it to be a serious problem.
>> E.g. when people complain about Emacs using a lot of memory it's always
>> (until now and I think in Nix's case as well) a case of too much memory
>> being used because of something like a leak, and not just "Emacs doesn't
>> return its free memory to the OS".
>
> It is a problem for everybody like me who doesn't have
> gigs of free RAM. Sorry.
>
> I'll remember to --disable-gnutls and not use Gnus for my
> own safety, for the moment. That's a good catch and seems
> to help.
Actually, Gnus is not a memory hog as long as you build Emacs
"--without-gnutls". But with gnutls support, it's clearly leaking
memory.
Cheers,
Sven
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: Emacs bzr memory footprint
2011-10-22 8:44 ` Emacs bzr memory footprint Sven Joachim
@ 2011-10-22 11:57 ` Carsten Mattner
0 siblings, 0 replies; 113+ messages in thread
From: Carsten Mattner @ 2011-10-22 11:57 UTC (permalink / raw)
To: Sven Joachim; +Cc: Dmitry Antipov, Stefan Monnier, emacs-devel
On Sat, Oct 22, 2011 at 10:44 AM, Sven Joachim <svenjoac@gmx.de> wrote:
> Actually, Gnus is not a memory hog as long as you build Emacs
> "--without-gnutls". But with gnutls support, it's clearly leaking
> memory.
Ah, I see. Thanks for the clarification, Sven.
Both gnus and erc-tls seem to also work with a forked openssl or gnutls
process instead of the linked-in gnutls lib, if I needed.
So I think I will avoid linking gnutls for the moment.
I have switched to irssi for IRC anyway.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-22 7:34 ` Carsten Mattner
2011-10-22 8:30 ` Stephen J. Turnbull
2011-10-22 8:44 ` Emacs bzr memory footprint Sven Joachim
@ 2011-10-23 18:04 ` Stefan Monnier
2 siblings, 0 replies; 113+ messages in thread
From: Stefan Monnier @ 2011-10-23 18:04 UTC (permalink / raw)
To: Carsten Mattner; +Cc: Dmitry Antipov, emacs-devel
> It is a problem for everybody like me who doesn't have
> gigs of free RAM. Sorry.
No, it's not. You misunderstand. The problem that I say is not
important is the fact that if ever Emacs gets up to N megabytes for good
reasons (you really use up that much stuff at a given time), and later
on you use up less stuff, the Emacs process may keep using up
N megabytes (most of it inactive).
If you have little memory and N megabytes is a problem, then it will be
a problem already when you do use that much, i.e. before the "waste"
comes into play.
FWIW, my office desktop is a 1GB fit-pc2.
> I'll remember to --disable-gnutls and not use Gnus for my own safety,
> for the moment. That's a good catch and seems to help.
Looks like we have a real leak there, yes.
Stefan
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: Emacs bzr memory footprint
2011-10-21 13:30 ` Stefan Monnier
2011-10-21 15:02 ` Nix
2011-10-21 15:34 ` Dmitry Antipov
@ 2011-10-25 17:35 ` Nix
2011-10-25 20:23 ` Stefan Monnier
2 siblings, 1 reply; 113+ messages in thread
From: Nix @ 2011-10-25 17:35 UTC (permalink / raw)
To: Stefan Monnier; +Cc: John Wiegley, emacs-devel
On 21 Oct 2011, Stefan Monnier uttered the following:
> I've just installed memory-usage.el as a package in the `elpa' branch
> (you'll soon be able to install it using package.el and in the mean time
> you can download it from http://bzr.savannah.gnu.org/lh/emacs/elpa/download/head:/memoryusage.el-20111021130523-o1v7kfuzrat3pcd9-2/memory-usage.el).
>
> It basically looks at the garbage-collect output and the various
> buffer's size to provide you with a human-readable description of the
> memory in use, from Elisp's point of view. Don't expect too much of it,
> but it can be helpful.
>
> Could you try it out?
Two rounds of stats. Firstly, absolute peak legitimate memory usage,
when reading a truly massive nonexpired newsgroup:
Garbage collection stats:
((1682002 . 1142162) (78246 . 155) (25344 . 17561) 88139150 2009895 (2439 . 9353) (32882 . 12331) (452963 . 384130))
=> 26912032+18274592 bytes in cons cells
3755808+7440 bytes in symbols
1013760+702440 bytes in markers
39024+149648 bytes in floats
1841392+690536 bytes in intervals
14494816+12292160 bytes in string headers
88139150 bytes of string chars
88139150 bytes of vector slots
Total bytes in lisp objects: 170322693 (live 138205877, dead 32116816)
Buffer ralloc memory usage:
108 buffers
7086099 bytes total (60682 in gaps)
[...]
In normal usage, with Huge Newsgroup closed but with my usual pair of
nnml and gnutls-backed nnimap groups open (after another GC to push
memory usage down properly):
Garbage collection stats:
((1678832 . 1131753) (77833 . 463) (24737 . 17683) 88021904 2005723 (2428 . 9364) (32180 . 14186) (449983 . 387110))
=> 26861312+18108048 bytes in cons cells
3735984+22224 bytes in symbols
989480+707320 bytes in markers
38848+149824 bytes in floats
1802080+794416 bytes in intervals
14399456+12387520 bytes in string headers
88021904 bytes of string chars
88021904 bytes of vector slots
Total bytes in lisp objects: 170024139 (live 137854787, dead 32169352)
Buffer ralloc memory usage:
104 buffers
6977389 bytes total (55669 in gaps)
Size Gap Name
STIME TIME RSS VSZ
Oct21 00:55:23 524568 716588
So I have 170Mb in Lisp objects, 7Mb in buffers... and a 716Mb VSZ.
500Mb of that is heap, too:
% pmap 2676 | grep anon
0000000002fc2000 499180K rw--- [ anon ]
00007fcd0017b000 4K rw--- [ anon ]
00007fcd05fbf000 4K rw--- [ anon ]
00007fcd08d72000 304K rw--- [ anon ]
00007fcd08e68000 348K rw--- [ anon ]
00007fcd094d1000 1300K rw--- [ anon ]
00007fcd097c6000 4K rw--- [ anon ]
00007fcd0ab36000 96K rw--- [ anon ]
00007fcd0ab9c000 436K rw--- [ anon ]
00007fcd0ac12000 424K rw--- [ anon ]
00007fcd0ac81000 192K rw--- [ anon ]
00007fcd0adea000 152K rw--- [ anon ]
00007fcd0ae23000 256K rw--- [ anon ]
00007fcd0aef9000 84K rw--- [ anon ]
00007fcd0dd80000 4K rw--- [ anon ]
00007fcd0e975000 4K rw--- [ anon ]
00007fcd0f1d7000 8K rw--- [ anon ]
00007fcd0f8db000 12K rw--- [ anon ]
00007fcd10ef8000 20K rw--- [ anon ]
00007fcd12041000 4K rw--- [ anon ]
00007fcd1225a000 16K rw--- [ anon ]
00007fcd12919000 100K rw--- [ anon ]
00007fcd12f1d000 4K rw--- [ anon ]
00007fcd1315e000 4K rw--- [ anon ]
00007fcd13672000 8K rw--- [ anon ]
00007fcd14df3000 64K rw--- [ anon ]
00007fcd15280000 16K rw--- [ anon ]
00007fcd156f1000 4K rw--- [ anon ]
00007fcd15b68000 8K rw--- [ anon ]
00007fcd15d38000 112K rw--- [ anon ]
00007fcd15d85000 8K rw--- [ anon ]
00007fcd15d89000 4K rw--- [ anon ]
00007fffca9ff000 4K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
Looky looky, leak.
> If this output does not explain the process's size you're seeing, then
> we have a leak in the C code somewhere.
Oh yes.
> I'd hack src/alloc.c to export the needed info to Elisp. But maybe
> memory-usage will already give us enough info that this won't
> be necessary.
Here's the last summary line from Dmitry Antipov's patch written for
exactly this purpose, which emits a *lot* of other usage info too
(if you want a 5Mb upload I can give you the whole log, lzmaed, but
look out, uncompressed it comes to 264M):
-- GC507 end: lisp data 179658 Kb (148315 live, 31342 free), 152325 mem
nodes (7140 Kb), heap 418617 Kb used, 117794 Kb free
That comes from this printf():
+ fprintf (stderr,
+ "-- GC%d end: lisp data %d Kb (%d live, %d free), "
+#if GC_MARK_STACK || defined GC_MALLOC_CHECK
+ "%d mem nodes (%d Kb), "
+#endif
+ "heap %d Kb used, %d Kb free\n",
+ /* GC count. */
+ gcs_done,
+ /* Total Lisp data. */
+ ((total_conses + total_free_conses) * sizeof (struct Lisp_Cons) +
+ (total_markers + total_free_markers) * sizeof (union Lisp_Misc) +
+ (total_symbols + total_free_symbols) * sizeof (struct Lisp_Symbol) +
+ total_string_size +
+ total_vector_size * sizeof (Lisp_Object) +
+ (total_floats + total_free_floats) * sizeof (struct Lisp_Float) +
+ (total_intervals + total_free_intervals) * sizeof (struct interval) +
+ (total_strings + total_free_strings) * sizeof (struct Lisp_String)) / 1024,
+ /* Live Lisp data. */
+ (total_conses * sizeof (struct Lisp_Cons) +
+ total_markers * sizeof (union Lisp_Misc) +
+ total_symbols * sizeof (struct Lisp_Symbol) +
+ total_string_size +
+ total_vector_size * sizeof (Lisp_Object) +
+ total_floats * sizeof (struct Lisp_Float) +
+ total_intervals * sizeof (struct interval) +
+ total_strings * sizeof (struct Lisp_String)) / 1024,
+ /* Free Lisp data. */
+ (total_free_conses * sizeof (struct Lisp_Cons) +
+ total_free_markers * sizeof (union Lisp_Misc) +
+ total_free_symbols * sizeof (struct Lisp_Symbol) +
+ total_free_floats * sizeof (struct Lisp_Float) +
+ total_free_intervals * sizeof (struct interval) +
+ total_free_strings * sizeof (struct Lisp_String)) / 1024,
+#if GC_MARK_STACK || defined GC_MALLOC_CHECK
+ total_mem_nodes, total_mem_nodes * sizeof (struct mem_node) / 1024,
+#endif
+#ifdef DOUG_LEA_MALLOC
+ mallinfo ().uordblks / 1024, mallinfo ().fordblks / 1024);
+#else
+ 0, 0);
+#endif
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: Emacs bzr memory footprint
2011-10-25 17:35 ` Nix
@ 2011-10-25 20:23 ` Stefan Monnier
2011-10-25 20:34 ` Nix
2011-10-26 14:52 ` gnutls memory leak [Was: Re: Emacs bzr memory footprint] Chong Yidong
0 siblings, 2 replies; 113+ messages in thread
From: Stefan Monnier @ 2011-10-25 20:23 UTC (permalink / raw)
To: Nix; +Cc: John Wiegley, emacs-devel
> So I have 170Mb in Lisp objects, 7Mb in buffers... and a 716Mb VSZ.
> 500Mb of that is heap, too:
>> If this output does not explain the process's size you're seeing, then
>> we have a leak in the C code somewhere.
> Oh yes.
Do you use the libgnutls bindings, by any chance?
If so, can you try to avoid using it and just stick to
gnutls-cli instead?
Stefan
^ permalink raw reply [flat|nested] 113+ messages in thread* Re: Emacs bzr memory footprint
2011-10-25 20:23 ` Stefan Monnier
@ 2011-10-25 20:34 ` Nix
2011-10-26 14:52 ` gnutls memory leak [Was: Re: Emacs bzr memory footprint] Chong Yidong
1 sibling, 0 replies; 113+ messages in thread
From: Nix @ 2011-10-25 20:34 UTC (permalink / raw)
To: Stefan Monnier; +Cc: John Wiegley, emacs-devel
On 25 Oct 2011, Stefan Monnier stated:
>>> If this output does not explain the process's size you're seeing, then
>>> we have a leak in the C code somewhere.
>> Oh yes.
>
> Do you use the libgnutls bindings, by any chance?
Yes.
> If so, can you try to avoid using it and just stick to
> gnutls-cli instead?
It looks like doing that is a simple matter of compiling it out and
open-network-stream will fall back automatically, so, sure, I'll do
that next time I rebuild (probably tomorrow), let it run for a few
days, and see if it bloats.
--
NULL && (void)
^ permalink raw reply [flat|nested] 113+ messages in thread
* gnutls memory leak [Was: Re: Emacs bzr memory footprint]
2011-10-25 20:23 ` Stefan Monnier
2011-10-25 20:34 ` Nix
@ 2011-10-26 14:52 ` Chong Yidong
2011-10-27 6:10 ` Chong Yidong
2011-10-27 22:03 ` Ted Zlatanov
1 sibling, 2 replies; 113+ messages in thread
From: Chong Yidong @ 2011-10-26 14:52 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Nix, John Wiegley, emacs-devel
I think I see a problem in gnutls.c. The functions for freeing the
credentials structures,
fn_gnutls_certificate_free_credentials (x509_cred);
fn_gnutls_anon_free_client_credentials (anon_cred);
are only called at gnutls.c:761 and gnutls.c:767, specifically for the
case where Fgnutls_boot is called on a process that already has
credentials structures (gnutls_x509_cred or gnutls_anon_cred) allocated.
IIUC, those freeing functions don't get called when the process is
killed via kill-process.
Could someone who had a hand in gnutls.c check if this is accurate? If
not, I can try to put together a patch.
BTW, does anyone know a simple test-case for using the gnutls support,
to make it easier to test for the leak?
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]
2011-10-26 14:52 ` gnutls memory leak [Was: Re: Emacs bzr memory footprint] Chong Yidong
@ 2011-10-27 6:10 ` Chong Yidong
2011-10-27 22:01 ` Ted Zlatanov
2011-10-27 22:03 ` Ted Zlatanov
1 sibling, 1 reply; 113+ messages in thread
From: Chong Yidong @ 2011-10-27 6:10 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Nix, John Wiegley, emacs-devel
Since no one responded so far, I went ahead and committed a fix for that
particular memory leak. I verified that with this fix, the credentials
structures are deallocated after sending a mail via SSL; previously they
won't.
If someone could check whether the memory leak is gone or still
persists, that would be great.
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]
2011-10-27 6:10 ` Chong Yidong
@ 2011-10-27 22:01 ` Ted Zlatanov
2011-10-28 12:04 ` Ted Zlatanov
0 siblings, 1 reply; 113+ messages in thread
From: Ted Zlatanov @ 2011-10-27 22:01 UTC (permalink / raw)
To: emacs-devel
On Thu, 27 Oct 2011 14:10:38 +0800 Chong Yidong <cyd@gnu.org> wrote:
CY> Since no one responded so far, I went ahead and committed a fix for that
CY> particular memory leak. I verified that with this fix, the credentials
CY> structures are deallocated after sending a mail via SSL; previously they
CY> won't.
CY> If someone could check whether the memory leak is gone or still
CY> persists, that would be great.
Thanks, I wasn't able to respond in time. But I doubt the credentials
are the problem. They don't get allocated/deallocated all that often,
unless I've misunderstood something fundamental in the GnuTLS code. I
suspect the error logging and general packet handling code.
Ted
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]
2011-10-27 22:01 ` Ted Zlatanov
@ 2011-10-28 12:04 ` Ted Zlatanov
2011-10-28 17:29 ` Nix
0 siblings, 1 reply; 113+ messages in thread
From: Ted Zlatanov @ 2011-10-28 12:04 UTC (permalink / raw)
To: emacs-devel
On Thu, 27 Oct 2011 18:01:47 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote:
TZ> On Thu, 27 Oct 2011 14:10:38 +0800 Chong Yidong <cyd@gnu.org> wrote:
CY> Since no one responded so far, I went ahead and committed a fix for that
CY> particular memory leak. I verified that with this fix, the credentials
CY> structures are deallocated after sending a mail via SSL; previously they
CY> won't.
CY> If someone could check whether the memory leak is gone or still
CY> persists, that would be great.
TZ> Thanks, I wasn't able to respond in time. But I doubt the credentials
TZ> are the problem. They don't get allocated/deallocated all that often,
TZ> unless I've misunderstood something fundamental in the GnuTLS code. I
TZ> suspect the error logging and general packet handling code.
After comparing an Emacs with and without your fix, actually, I see a
2MB difference in the memory usage every time a GnuTLS connection is
opened. That may have been the culprit, indeed.
I'm running the new Emacs and trying to balloon the memory usage but so
far have been unsuccessful.
Nix, Carsten, Sven, could you give a recent build a try?
Ted
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]
2011-10-28 12:04 ` Ted Zlatanov
@ 2011-10-28 17:29 ` Nix
0 siblings, 0 replies; 113+ messages in thread
From: Nix @ 2011-10-28 17:29 UTC (permalink / raw)
To: emacs-devel
On 28 Oct 2011, Ted Zlatanov said:
> Nix, Carsten, Sven, could you give a recent build a try?
Running 106207, started yesterday, used only for Gnus, I see:
Garbage collection stats:
((1750068 . 1022571) (81061 . 289) (17721 . 11917) 87140622 1911189 (1639 . 5992) (48873 . 14329) (459662 . 153301))
=> 28001088+16361136 bytes in cons cells
3890928+13872 bytes in symbols
708840+476680 bytes in markers
26224+95872 bytes in floats
2736888+802424 bytes in intervals
14709184+4905632 bytes in string headers
87140622 bytes of string chars
87140622 bytes of vector slots
Total bytes in lisp objects: 161780579 (live 139124963, dead 22655616)
Buffer ralloc memory usage:
107 buffers
8877238 bytes total (48855 in gaps)
USER RSS VSZ
nix 335476 627048
0000000001e19000 312492K rw--- [ anon ]
00007f3ab020e000 4K rw--- [ anon ]
00007f3ab4000000 132K rw--- [ anon ]
00007f3ab4021000 65404K ----- [ anon ]
00007f3ab839a000 4K rw--- [ anon ]
00007f3ab8aec000 56K rw--- [ anon ]
00007f3ab92c6000 1292K rw--- [ anon ]
00007f3ab940c000 4K rw--- [ anon ]
00007f3abac9c000 4K ----- [ anon ]
00007f3abac9d000 32768K rw--- [ anon ]
157988Kb + 8669Kb <<< 312492Kb (definitely the main heap), and this has
gone up by >150Mb since steady-state usage began yesterday.
The leak is still there, it seems.
I'll run it under valgrind and see what I can see.
--
NULL && (void)
^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: gnutls memory leak [Was: Re: Emacs bzr memory footprint]
2011-10-26 14:52 ` gnutls memory leak [Was: Re: Emacs bzr memory footprint] Chong Yidong
2011-10-27 6:10 ` Chong Yidong
@ 2011-10-27 22:03 ` Ted Zlatanov
1 sibling, 0 replies; 113+ messages in thread
From: Ted Zlatanov @ 2011-10-27 22:03 UTC (permalink / raw)
To: emacs-devel
On Wed, 26 Oct 2011 22:52:21 +0800 Chong Yidong <cyd@gnu.org> wrote:
CY> BTW, does anyone know a simple test-case for using the gnutls support,
CY> to make it easier to test for the leak?
gnutls.el has some in the comments:
;; (open-gnutls-stream "tls" "tls-buffer" "yourserver.com" "https")
;; (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
Then you can just kill that buffer.
Ted
^ permalink raw reply [flat|nested] 113+ messages in thread