command and doco for valgrind

command and doco for valgrind

[Nick Roberts]

Here's a patch for valgrind relating to my earlier message. Any comments?

Nick


--- gud.el	2004-01-09 21:14:21.000000000 +0000
***************
*** 543,548 ****
--- 543,553 ----
    (setq gdb-first-prompt t)
    (run-hooks 'gdb-mode-hook))
  
+ (defun valgrind (command-line)
+   (interactive (list
+    (read-string "Run valgrind (like this): " "valgrind --gdb-attach=yes ")))
+    (gdb command-line))
+ 
  ;; One of the nice features of GDB is its impressive support for
  ;; context-sensitive command completion.  We preserve that feature
  ;; in the GUD buffer by using a GDB command designed just for Emacs.



*** building.texi.~1.34.~	2003-12-18 21:44:37.000000000 +0000
--- building.texi	2004-01-09 21:37:09.000000000 +0000
***************
*** 300,305 ****
--- 300,306 ----
  * GDB Graphical Interface::  An enhanced mode that uses GDB features to
                          implement a graphical debugging environment through
                          Emacs.
+ * Using Valgrind::       Memory management in the GUD buffer.
  @end menu
  
  @node Starting GUD
***************
*** 798,803 ****
--- 799,845 ----
  gdb-frame-@var{buffertype}-buffer} respectively, where @var{buffertype}
  is the relevant buffer type e.g breakpoints.
  
+ @node Using Valgrind
+ @subsection Using Valgrind in the GUD buffer
+ @cindex Valgrind
+ 
+ Valgrind is a tool to help you find memory-management problems in your
+ programs.  You can use it from the command line and this section
+ descibes how to use it with the GUD buffer.
+ 
+ To start running your program through valgrind type:
+ 
+ @example
+ M-x valgrind @key{RET} @var{file} @key{RET}
+ @end example
+ 
+ At a memory violation, when valgrind asks if you want to attach to gdb, type y:
+ 
+ @example
+ ==19752== ---- Attach to GDB ? --- [Return/N/n/Y/y/C/c] ---- y
+ @end example
+ 
+ At the prompt for GDB type:
+ 
+ @example
+ @code{set ann 1} and type @key{RET} if you want the mode for M-x gdb.
+ @code{set ann 3} and type @key{RET} if you want the mode for M-x gdba.
+ @end example
+ 
+ In the first case, the main routine appears in the source buffer and
+ the resulting layout depends on the value of gdb-many-windows.  In the
+ second case, nothing happens immediately.
+ 
+ If you now type bt, GDB prints the call stack.  This also includes
+ calls to valgrind's code.  Identify the frame number of your code, 6
+ say, and type @code{frame 6} in the GUD buffer.  The source code for
+ this call should appear in another buffer in both cases.  Just as with
+ operation from the command line, you can't step through your code when
+ it is run through valgrind.  However you can move up and down the
+ stack and examine the values of variables.  When you want to return
+ control to valgrind type @kbd{C-d} to quit GDB but stay in the GUD
+ buffer.
+ 
  @node Executing Lisp
  @section Executing Lisp Expressions

Re: command and doco for valgrind

[Eli Zaretskii]

> From: Nick Roberts <nick@nick.uklinux.net>
> Date: Fri, 9 Jan 2004 21:42:15 +0000
> 
> Here's a patch for valgrind relating to my earlier message. Any comments?

This is IMHO a borderline case: not only do we document GDB-related
features in the Emacs manual, but we also document usage of another
program that might help debugging programs.  I dunno.

Anyway, assuming that we do include this in the manual, here are some
comments regarding Texinfo usage:

> + @cindex Valgrind

Index entries regarding programs should use @pindex, not @cindex.

> + To start running your program through valgrind type:

"valgrind" should have the @code markup, since it's a program name.

> + At a memory violation, when valgrind asks if you want to attach to gdb, type y:

Ditto, and the same applies to "gdb".

> + At the prompt for GDB type:
> + 
> + @example
> + @code{set ann 1} and type @key{RET} if you want the mode for M-x gdb.
> + @code{set ann 3} and type @key{RET} if you want the mode for M-x gdba.
> + @end example

I think you should use @kbd here, like this:

  At GDB's prompt type @kbd{set ann 1 @key{RET}} if you want the mode
  for @samp{M-x gdb}, or @kbd{set ann 3 @key{RET}} if you want the mode
  for @samp{M-x gdba}.

> + the resulting layout depends on the value of gdb-many-windows.  In the

"gdb-many-windows" is a variable, so its name should have the @code
markup.

> + If you now type bt, GDB prints the call stack.  This also includes

"bt" is something the user types, so it should have the @kbd markup.

Re: command and doco for valgrind

[Nick Roberts]

 > > Here's a patch for valgrind relating to my earlier message. Any comments?
 > 
 > This is IMHO a borderline case: not only do we document GDB-related
 > features in the Emacs manual, but we also document usage of another
 > program that might help debugging programs.  I dunno.

I got no postive feedback the valgrind users list either so I won't commit
this patch. If I can develop the idea further, I will re-post it. Thanks for
an honest reply.

BTW I think valgrind could be a useful tool to debug some of Emacs crashes
(less frequent now) on x86 architectures. However, for some reason, it reports
a memory full message at start up, at the moment:

==2466== Invalid free() / delete / delete[]
==2466==    at 0x400263A8: free (vg_replace_malloc.c:231)
==2466==    by 0x812A135: memory_full (alloc.c:478)
==2466==    by 0x812A3E8: lisp_malloc (alloc.c:623)
==2466==    by 0x812BA07: allocate_vectorlike (alloc.c:2505)
==2466==    Address 0x83CCE60 is not stack'd, malloc'd or free'd
emacs: Memory exhausted--use M-x save-some-buffers then exit and restart Emacs

Nick

Re: command and doco for valgrind

[Eli Zaretskii]

> Date: Sat, 17 Jan 2004 23:38:58 +0000
> From: Nick Roberts <nickrob@gnu.org>
> 
> BTW I think valgrind could be a useful tool to debug some of Emacs crashes
> (less frequent now) on x86 architectures. However, for some reason, it reports
> a memory full message at start up, at the moment:
> 
> ==2466== Invalid free() / delete / delete[]
> ==2466==    at 0x400263A8: free (vg_replace_malloc.c:231)
> ==2466==    by 0x812A135: memory_full (alloc.c:478)
> ==2466==    by 0x812A3E8: lisp_malloc (alloc.c:623)
> ==2466==    by 0x812BA07: allocate_vectorlike (alloc.c:2505)
> ==2466==    Address 0x83CCE60 is not stack'd, malloc'd or free'd
> emacs: Memory exhausted--use M-x save-some-buffers then exit and restart Emacs

I suspect that it has something to do with the dumping process (which
effectively freezes memory malloc'ed before unexec), but I don't know
enough about valgrind to tell for sure.

Re: command and doco for valgrind

[Eli Zaretskii]

> Date: Sat, 17 Jan 2004 23:38:58 +0000
> From: Nick Roberts <nickrob@gnu.org>
> 
> BTW I think valgrind could be a useful tool to debug some of Emacs crashes
> (less frequent now) on x86 architectures. However, for some reason, it reports
> a memory full message at start up, at the moment:
> 
> ==2466== Invalid free() / delete / delete[]
> ==2466==    at 0x400263A8: free (vg_replace_malloc.c:231)
> ==2466==    by 0x812A135: memory_full (alloc.c:478)
> ==2466==    by 0x812A3E8: lisp_malloc (alloc.c:623)
> ==2466==    by 0x812BA07: allocate_vectorlike (alloc.c:2505)
> ==2466==    Address 0x83CCE60 is not stack'd, malloc'd or free'd
> emacs: Memory exhausted--use M-x save-some-buffers then exit and restart Emacs

FWIW, this problem is documented in the valgrind's docs.

I'll try to look into this and see what I find.

Re: command and doco for valgrind

[Richard Stallman]

I'm not too surprised that unexec runs into trouble with valgrind.
unexec is not a 100% implementation of undumping; it handles only the
cases that people needed to make it handle.  If valgrind moves outside
that envelope, problems occur.

It may be possible to debug the problem by running temacs with args -l
loadup.  That way you can do the debugging without ever dumping Emacs.

Re: command and doco for valgrind

[Eli Zaretskii]

> Date: 18 Jan 2004 16:17:11 +0200
> From: Eli Zaretskii <eliz@elta.co.il>
> 
> > Date: Sat, 17 Jan 2004 23:38:58 +0000
> > From: Nick Roberts <nickrob@gnu.org>
> > 
> > BTW I think valgrind could be a useful tool to debug some of Emacs crashes
> > (less frequent now) on x86 architectures. However, for some reason, it reports
> > a memory full message at start up, at the moment:
> > 
> > ==2466== Invalid free() / delete / delete[]
> > ==2466==    at 0x400263A8: free (vg_replace_malloc.c:231)
> > ==2466==    by 0x812A135: memory_full (alloc.c:478)
> > ==2466==    by 0x812A3E8: lisp_malloc (alloc.c:623)
> > ==2466==    by 0x812BA07: allocate_vectorlike (alloc.c:2505)
> > ==2466==    Address 0x83CCE60 is not stack'd, malloc'd or free'd
> > emacs: Memory exhausted--use M-x save-some-buffers then exit and restart Emacs
> 
> FWIW, this problem is documented in the valgrind's docs.
> 
> I'll try to look into this and see what I find.

After futzing with this for a while, all I can say is "weird".  The
above problem happens because lisp_malloc calls malloc to allocate a
Lisp vector during startup, and malloc returns NULL.  Emacs then
rightfully decides that there's not enough memory and that therefore
it cannot continue its startup, and it exits.  (The mesage about
free'in memory that wasn't malloc'ed seems to be a separate problem.)

Why malloc returns NULL is a mystery.  Emacs on GNU/Linux (the only
platform supported by valgrind) uses malloc from glibc, so I cannot
easily look into that, as I don't have the sources that correspond to
the particular version of glibc used by that box (fencepost.gnu.org, a
Debian system).  Perhaps someone who has the sources could do that.

Moreover, even temacs appears to crash under valgrind: try

  valgrind ./temacs

or

  valgrind ./temacs -l loadup

They both crash with SIGSEGV in some way or another.  "./temacs"
without valgrind works as expected, of course.

Did I say it's weird?

I think someone who knows more about valgrind's messing with the
instruction stream should debug this.  Since this problem happens only
under valgrind, and valgrind doesn't run the original code of the
Emacs binary, but instead modifies the instruction stream in many ways
and executes the modified code on a synthetic CPU that is part of
valgrind's core, it's not even clear to me how to debug such problems
with GDB.

Sorry.

Re: command and doco for valgrind

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sun, 18 Jan 2004 14:15:13 -0500
> 
> It may be possible to debug the problem by running temacs with args -l
> loadup.

As I said in a later message, I already tried that: it crashes with
SIGSEGV.

Re: command and doco for valgrind

[Richard Stallman]

    Moreover, even temacs appears to crash under valgrind: try

      valgrind ./temacs

    or

      valgrind ./temacs -l loadup

    They both crash with SIGSEGV in some way or another.  "./temacs"
    without valgrind works as expected, of course.

This could be a completely different problem.
So how about looking at this first?

I can guess that the problem could be due to the fact that Emacs uses
its own malloc.

Re: command and doco for valgrind

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 19 Jan 2004 15:11:41 -0500
> 
>       valgrind ./temacs
> 
>     or
> 
>       valgrind ./temacs -l loadup
> 
>     They both crash with SIGSEGV in some way or another.  "./temacs"
>     without valgrind works as expected, of course.
> 
> This could be a completely different problem.

Maybe, I don't know.  FWIW, this is the first time I see temacs crash
like that.

> So how about looking at this first?

I cannot figure out how; suggestions welcome.

The problem is that valgrind doesn't really run the machine
instructions from temacs on the real CPU.  Instead, it runs a modified
instrcution stream (modifications are its instrumentation of the code)
on a synthetic CPU provided by the valgrind core.  Given this, I can't
seem to find a way to run "valgrind ./temacs" under GDB, and without
that, how can I debug this?

> I can guess that the problem could be due to the fact that Emacs uses
> its own malloc.

Most probably, although on GNU/Linux we use the one from glibc, IIRC
(which is Doug Lea's variant).

Re: command and doco for valgrind

[Nick Roberts]

 > After futzing with this for a while, all I can say is "weird".  The
 > above problem happens because lisp_malloc calls malloc to allocate a
 > Lisp vector during startup, and malloc returns NULL.  Emacs then
 > rightfully decides that there's not enough memory and that therefore
 > it cannot continue its startup, and it exits.

Are you sure malloc returns NULL? Later you say:

> Given this, I can't seem to find a way to run "valgrind ./temacs" under GDB,
> and without that, how can I debug this?

I guess you've tried variants of "valgrind --gdb-attach=yes emacs". Thats what
my original post was about :-)

The problem, here, is that gdb attaches to emacs after it has called
memory_full.  However, if I do p lisp_malloc_loser, I get something like:

$1 = (void *) 0x418aad88

so it looks like val is *set* from this value to 0 in the loop:

      if ((char *) XCONS (tem) != (char *) val + nbytes - 1)
	{
	  lisp_malloc_loser = val;
	  free (val);
	  val = 0;
	}

I don't know about x86 internals but is this away from the heap and somewhere
where shared libraries go? If I debug emacs directly, I get a lower address
value for val:

$1 = (void *) 0x8641e58


    Nick                                         http://www.nick.uklinux.net

Re: EMACS and valgrind (was: command and doco for valgrind)

[Eli Zaretskii]

[Let's start a new thread for this.]

> Date: Mon, 19 Jan 2004 22:57:17 +0000
> From: Nick Roberts <nickrob@gnu.org>
> 
> > Given this, I can't seem to find a way to run "valgrind ./temacs" under GDB,
> > and without that, how can I debug this?
> 
> I guess you've tried variants of "valgrind --gdb-attach=yes emacs". Thats what
> my original post was about :-)
> 
> The problem, here, is that gdb attaches to emacs after it has called
> memory_full.

Exactly.  Moreover, the code that just caused the problem is not the
code GDB will see.

> However, if I do p lisp_malloc_loser, I get something like:
> 
> $1 = (void *) 0x418aad88
> 
> so it looks like val is *set* from this value to 0 in the loop:
> 
>       if ((char *) XCONS (tem) != (char *) val + nbytes - 1)
> 	{
> 	  lisp_malloc_loser = val;
> 	  free (val);
> 	  val = 0;
> 	}

Yes, you are right, I think.

> I don't know about x86 internals but is this away from the heap and somewhere
> where shared libraries go?

I don't know either.  Anyone?

Re: EMACS and valgrind (was: command and doco for valgrind)

[Stefan Monnier]

>> I don't know about x86 internals but is this away from the heap and
>> somewhere where shared libraries go?
> I don't know either.  Anyone?

Well, with my USE_LSB_TAG patch, you might be able to go a bit further
since it does not assume that malloc returns blocks of memory close to
address 0 (instead it assumes the blocks are aligned on 8byte boundaries).

It's almost ready for installation.


        Stefan

Re: EMACS and valgrind

[Nick Roberts]

 > Well, with my USE_LSB_TAG patch, you might be able to go a bit further
 > since it does not assume that malloc returns blocks of memory close to
 > address 0 (instead it assumes the blocks are aligned on 8byte boundaries).

Well, I've rebuilt Emacs with make "MYCPPFLAGS=-DUSE_LSB_TAG". Is that right?
Valgrind complains that the memory is exhausted again, but from a different
location in the Emacs code (shown below). To give me a handle on this matter,
can you please provide a bit of background to this patch?

Nick


==2395== Memcheck, a.k.a. Valgrind, a memory error detector for x86-linux.
==2395== Copyright (C) 2002-2003, and GNU GPL'd, by Julian Seward.
==2395== Using valgrind-2.0.0, a program supervision framework for x86-linux.
==2395== Copyright (C) 2000-2003, and GNU GPL'd, by Julian Seward.
==2395== Estimated CPU clock rate is 200 MHz
==2395== For more details, rerun with: -v
==2395== 
==2395== Invalid free() / delete / delete[]
==2395==    at 0x4002668F: realloc (vg_replace_malloc.c:310)
==2395==    by 0x812E694: xrealloc (alloc.c:547)
==2395==    by 0x81267BE: regex_compile (regex.c:2721)
==2395==    by 0x812D8BF: re_compile_pattern (regex.c:5997)
==2395==    Address 0x8575778 is not stack'd, malloc'd or free'd
==2395== 
==2395== Invalid free() / delete / delete[]
==2395==    at 0x400263A8: free (vg_replace_malloc.c:231)
==2395==    by 0x812E595: memory_full (alloc.c:478)
==2395==    by 0x812E6D8: xrealloc (alloc.c:552)
==2395==    by 0x81267BE: regex_compile (regex.c:2721)
==2395==    Address 0x83D2F60 is not stack'd, malloc'd or free'd
emacs: Memory exhausted--use M-x save-some-buffers then exit and restart Emacs

Re: EMACS and valgrind

[Stefan Monnier]

>> Well, with my USE_LSB_TAG patch, you might be able to go a bit further
>> since it does not assume that malloc returns blocks of memory close to
>> address 0 (instead it assumes the blocks are aligned on 8byte boundaries).

> Well, I've rebuilt Emacs with make "MYCPPFLAGS=-DUSE_LSB_TAG". Is that right?
> Valgrind complains that the memory is exhausted again, but from a different
> location in the Emacs code (shown below). To give me a handle on this matter,
> can you please provide a bit of background to this patch?

The patch changes the place where tag bits are kept in Lisp_Object values.
I.e. instead of using the top 3 bits (so we can only use pointers whose
values are smaller than 2^29) it now uses the low 3 bits (so we can use
pointers that point anywhere within the address space so long as they are
aligned on an 8-byte boundary).

Now, I have no idea whether the problem you se without USE_LSB_TAG is due to
the fact that valgrind uses a malloc implementation that returns pointers
whose value is larger than 2^29, but it seemed like a possible cause.

> ==2395== Memcheck, a.k.a. Valgrind, a memory error detector for x86-linux.
> ==2395== Copyright (C) 2002-2003, and GNU GPL'd, by Julian Seward.
> ==2395== Using valgrind-2.0.0, a program supervision framework for x86-linux.
> ==2395== Copyright (C) 2000-2003, and GNU GPL'd, by Julian Seward.
> ==2395== Estimated CPU clock rate is 200 MHz
> ==2395== For more details, rerun with: -v
> ==2395== 
> ==2395== Invalid free() / delete / delete[]
> ==2395==    at 0x4002668F: realloc (vg_replace_malloc.c:310)
> ==2395==    by 0x812E694: xrealloc (alloc.c:547)
> ==2395==    by 0x81267BE: regex_compile (regex.c:2721)
> ==2395==    by 0x812D8BF: re_compile_pattern (regex.c:5997)
> ==2395==    Address 0x8575778 is not stack'd, malloc'd or free'd
> ==2395== 
> ==2395== Invalid free() / delete / delete[]
> ==2395==    at 0x400263A8: free (vg_replace_malloc.c:231)
> ==2395==    by 0x812E595: memory_full (alloc.c:478)
> ==2395==    by 0x812E6D8: xrealloc (alloc.c:552)
> ==2395==    by 0x81267BE: regex_compile (regex.c:2721)
> ==2395==    Address 0x83D2F60 is not stack'd, malloc'd or free'd
> emacs: Memory exhausted--use M-x save-some-buffers then exit and restart Emacs

This does not look like exhausted memory but rather looks like corruption
(c.f. "Invalid free").


        Stefan

Re: EMACS and valgrind

[Nick Roberts]

 > Now, I have no idea whether the problem you se without USE_LSB_TAG is due to
 > the fact that valgrind uses a malloc implementation that returns pointers
 > whose value is larger than 2^29, but it seemed like a possible cause.

 It looks like USE_LSB_TAG has moved the problem from a lisp one to one of
 memory allocation. As Eli has said earlier, the valgrind team have noted that
 memory debugging doesn't work with Emacs:

doc> Programs which are known not to work are:

doc>    * emacs starts up but immediately concludes it is out of memory and
doc>    aborts. Emacs has it's own memory-management scheme, but I don't
doc>    understand why this should interact so badly with Valgrind. Emacs
doc>    works fine if you build it to use the standard malloc/free routines.

So I guess its not a simple problem.


Nick

Re: EMACS and valgrind

[Eli Zaretskii]

> From: Nick Roberts <nick@nick.uklinux.net>
> Date: Fri, 23 Jan 2004 22:07:44 +0000
> 
>  It looks like USE_LSB_TAG has moved the problem from a lisp one to one of
>  memory allocation. As Eli has said earlier, the valgrind team have noted that
>  memory debugging doesn't work with Emacs:
> 
> doc> Programs which are known not to work are:
> 
> doc>    * emacs starts up but immediately concludes it is out of memory and
> doc>    aborts. Emacs has it's own memory-management scheme, but I don't
> doc>    understand why this should interact so badly with Valgrind. Emacs
> doc>    works fine if you build it to use the standard malloc/free routines.
> 
> So I guess its not a simple problem.

I think we need to bring the valgrind developer(s) on board to help
debugging this.  Perhaps Richard could ask him/her/them to help.
valgrind sounds like a great tool for finding memory-related bugs;
it'd be a pity if we couldn't find a way to use it.

Note that there's a section in etc/DEBUG about building Emacs with
malloc-debugging packages (most of the info there is actually based on
Purify, but the name isn't mentioned because Purify is not Free
Software).  It's possible that some of the advice there could help
with valgrind, although I wouldn't hold my breath, since valgrind
inner workings are very different, AFAICT.

As another note, the above citation claims that Emacs compiled with
``the standard malloc'' works fine under valgrind, but I always
thought that Emacs built on GNU/Linux uses the system malloc by
default, isn't that so?

Re: command and doco for valgrind

[Nick Roberts]

 > > BTW I think valgrind could be a useful tool to debug some of Emacs crashes
 > > (less frequent now) on x86 architectures. However, for some reason, it reports
 > > a memory full message at start up, at the moment:
 > > 
 > > ==2466== Invalid free() / delete / delete[]
 > > ==2466==    at 0x400263A8: free (vg_replace_malloc.c:231)
 > > ==2466==    by 0x812A135: memory_full (alloc.c:478)
 > > ==2466==    by 0x812A3E8: lisp_malloc (alloc.c:623)
 > > ==2466==    by 0x812BA07: allocate_vectorlike (alloc.c:2505)
 > > ==2466==    Address 0x83CCE60 is not stack'd, malloc'd or free'd
 > > emacs: Memory exhausted--use M-x save-some-buffers then exit and restart Emacs
 > 
 > FWIW, this problem is documented in the valgrind's docs.

How about including this in Emacs PROBLEMS file also?

Nick

Re: command and doco for valgrind

[Eli Zaretskii]

> From: Nick Roberts <nick@nick.uklinux.net>
> Date: Sun, 21 Mar 2004 16:47:04 +0000
>  > 
>  > FWIW, this problem is documented in the valgrind's docs.
> 
> How about including this in Emacs PROBLEMS file also?

Do we have something useful to tell users, like how to work around the
problem?  If not, I personally don't see any reason to have this same
text in PROBLEMS.

Re: command and doco for valgrind

[Nick Roberts]

 > >  > FWIW, this problem is documented in the valgrind's docs.
 > > 
 > > How about including this in Emacs PROBLEMS file also?
 > 
 > Do we have something useful to tell users, like how to work around the
 > problem?  If not, I personally don't see any reason to have this same
 > text in PROBLEMS.

The reasoning would be that someone, on seeing it, might work on it (to Emacs
benefit). It's probably not appropriate for the TODO file, and like many
tasks, such as GNOME support for Emacs, it falls between the cracks of two
different projects and gets forgotten about. I'm not suggesting copying it
verbatim.

Nick