Possible memory corruption problem

Possible memory corruption problem

[Piet van Oostrum]

I have had a few occasions when my mailboxes got corrupted.
I read my normal email with VM, and mail from mailing lists with gnus. I
have CVS emacs compiled about a month ago, running on Mac OS X 10.4.4.

A few cases now, the last of which was last weekend, the mailboxes that I
had open got completely mixed up. I lost my INBOX (primary VM mailbox)
completely last night, and some of its messages where found in the middle
of another mailbox that I had open. The mixup also occurred around last
Christmas with VM and gnus mailboxes mixed up. Sometimes the corrupted
mailboxes appeared to contain garbage text from other files.

My mailboxes are usually the largest files that I have open in Emacs, a few
MB each.

I have tried to find the common circumstances when this happens and it
aeems to me that it happens when the machine is low on virtual memory
(including swap space). So I suspect that some part of Emacs' buffer and
memory management fails to check the result of malloc calls or something
similar. I have written a test program and it appears that Mac OS X's
malloc and realloc correctly return NULL when the process runs out of
memory. 

I am sorry I can't give any more details at the moment. However I can add
that Emacs did not crash in the cases mentioned.
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: Possible memory corruption problem

[Eli Zaretskii]

> From: Piet van Oostrum <piet@cs.uu.nl>
> Date: Mon, 06 Feb 2006 16:36:01 +0100
> 
> I have had a few occasions when my mailboxes got corrupted.
> I read my normal email with VM, and mail from mailing lists with gnus. I
> have CVS emacs compiled about a month ago, running on Mac OS X 10.4.4.
> 
> A few cases now, the last of which was last weekend, the mailboxes that I
> had open got completely mixed up. I lost my INBOX (primary VM mailbox)
> completely last night, and some of its messages where found in the middle
> of another mailbox that I had open. The mixup also occurred around last
> Christmas with VM and gnus mailboxes mixed up. Sometimes the corrupted
> mailboxes appeared to contain garbage text from other files.

Sounds like some snafu with relocating buffers.  See below.

> I have tried to find the common circumstances when this happens and it
> aeems to me that it happens when the machine is low on virtual memory
> (including swap space).

Emacs should display a warning when the system is low on memory.  Does
it?

> So I suspect that some part of Emacs' buffer and memory management
> fails to check the result of malloc calls or something similar.

I'd rather suspect something else: that some library function on your
platform is passed a pointer to buffer text, and that library function
calls malloc internally, which (especially when Emacs needs to
allocate a large amount of memory) causes Emacs to relocate buffer
text, which could easily invalidate the pointer(s) used by that
library function.

> I have written a test program and it appears that Mac OS X's malloc
> and realloc correctly return NULL when the process runs out of
> memory.

Does Emacs indeed use system malloc on your platform?  Or does it use
gmalloc?

Re: Possible memory corruption problem

[Piet van Oostrum]

>>>>> Eli Zaretskii <eliz@gnu.org> (EZ) wrote:

>>> From: Piet van Oostrum <piet@cs.uu.nl>
>>> Date: Mon, 06 Feb 2006 16:36:01 +0100
>>> 
>>> I have had a few occasions when my mailboxes got corrupted.
>>> I read my normal email with VM, and mail from mailing lists with gnus. I
>>> have CVS emacs compiled about a month ago, running on Mac OS X 10.4.4.
>>> 
>>> A few cases now, the last of which was last weekend, the mailboxes that I
>>> had open got completely mixed up. I lost my INBOX (primary VM mailbox)
>>> completely last night, and some of its messages where found in the middle
>>> of another mailbox that I had open. The mixup also occurred around last
>>> Christmas with VM and gnus mailboxes mixed up. Sometimes the corrupted
>>> mailboxes appeared to contain garbage text from other files.

>EZ> Sounds like some snafu with relocating buffers.  See below.

>>> I have tried to find the common circumstances when this happens and it
>>> aeems to me that it happens when the machine is low on virtual memory
>>> (including swap space).

>EZ> Emacs should display a warning when the system is low on memory.  Does
>EZ> it?

I haven't found a message. On the other hand I found an older crash
dump with an alloc problem:

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_INVALID_ADDRESS (0x0001) at 0x80040907

Thread 0 Crashed:
0   org.gnu.Emacs 	0x000cc388 Fcons + 0x34 (alloc.c:2700)
1   org.gnu.Emacs 	0x000cc744 Fmake_list + 0xec (alloc.c:2829)
2   org.gnu.Emacs 	0x000eb6a0 concat + 0x370 (fns.c:656)
3   org.gnu.Emacs 	0x000ec684 Fcopy_alist + 0x78 (fns.c:1224)
4   org.gnu.Emacs 	0x00016d30 Fframe_parameters + 0x9c (frame.c:2090)
5   org.gnu.Emacs 	0x000172d0 Fframe_parameter + 0x230 (frame.c:2237)
6   org.gnu.Emacs 	0x000e6674 Ffuncall + 0x300 (eval.c:2866)
7   org.gnu.Emacs 	0x00113a2c Fbyte_code + 0x97c (bytecode.c:691)
8   org.gnu.Emacs 	0x000e6cb8 funcall_lambda + 0x2e8 (eval.c:3054)
9   org.gnu.Emacs 	0x000e67c4 Ffuncall + 0x450 (eval.c:2917)
10  org.gnu.Emacs 	0x00113a2c Fbyte_code + 0x97c (bytecode.c:691)
11  org.gnu.Emacs 	0x000e6cb8 funcall_lambda + 0x2e8 (eval.c:3054)
12  org.gnu.Emacs 	0x000e67c4 Ffuncall + 0x450 (eval.c:2917)
13  org.gnu.Emacs 	0x00113a2c Fbyte_code + 0x97c (bytecode.c:691)
14  org.gnu.Emacs 	0x000e6cb8 funcall_lambda + 0x2e8 (eval.c:3054)
15  org.gnu.Emacs 	0x000e67c4 Ffuncall + 0x450 (eval.c:2917)
16  org.gnu.Emacs 	0x00113a2c Fbyte_code + 0x97c (bytecode.c:691)
17  org.gnu.Emacs 	0x000e6cb8 funcall_lambda + 0x2e8 (eval.c:3054)
18  org.gnu.Emacs 	0x000e67c4 Ffuncall + 0x450 (eval.c:2917)
19  org.gnu.Emacs 	0x00113a2c Fbyte_code + 0x97c (bytecode.c:691)
20  org.gnu.Emacs 	0x000e6cb8 funcall_lambda + 0x2e8 (eval.c:3054)
21  org.gnu.Emacs 	0x000e67c4 Ffuncall + 0x450 (eval.c:2917)
22  org.gnu.Emacs 	0x00113a2c Fbyte_code + 0x97c (bytecode.c:691)
23  org.gnu.Emacs 	0x000e6cb8 funcall_lambda + 0x2e8 (eval.c:3054)
24  org.gnu.Emacs 	0x000e6968 apply_lambda + 0xfc (eval.c:2974)
25  org.gnu.Emacs 	0x000e5adc Feval + 0x564 (eval.c:2262)
26  org.gnu.Emacs 	0x000e2804 Fand + 0x48 (eval.c:354)
27  org.gnu.Emacs 	0x000e592c Feval + 0x3b4 (eval.c:2205)
28  org.gnu.Emacs 	0x000e4634 internal_condition_case_1 + 0x148 (eval.c:1494)
29  org.gnu.Emacs 	0x00083cc0 menu_item_eval_property + 0x7c (keyboard.c:7160)
30  org.gnu.Emacs 	0x00084db0 parse_tool_bar_item + 0x3e0 (keyboard.c:7837)
31  org.gnu.Emacs 	0x000849b0 process_tool_bar_item + 0xbc (keyboard.c:7667)
32  org.gnu.Emacs 	0x000848a4 tool_bar_items + 0x1e8 (keyboard.c:7619)
33  org.gnu.Emacs 	0x00028680 update_tool_bar + 0x1b4 (xdisp.c:8996)
34  org.gnu.Emacs 	0x00027fdc prepare_menu_bars + 0x170 (xdisp.c:8668)
35  org.gnu.Emacs 	0x0002aa74 redisplay_internal + 0x36c (xdisp.c:10384)
36  org.gnu.Emacs 	0x0002b9a0 redisplay_preserve_echo_area + 0x50 (xdisp.c:10989)
37  org.gnu.Emacs 	0x0011acfc wait_reading_process_output + 0x3c0 (process.c:4167)
38  org.gnu.Emacs 	0x00012eac sit_for + 0xcc (dispnew.c:6419)
39  org.gnu.Emacs 	0x0007e138 read_char + 0xa1c (keyboard.c:2765)
40  org.gnu.Emacs 	0x000863f4 read_key_sequence + 0x790 (keyboard.c:8829)
41  org.gnu.Emacs 	0x0007b8c8 command_loop_1 + 0x42c (keyboard.c:1529)
42  org.gnu.Emacs 	0x000e44c8 internal_condition_case + 0x140 (eval.c:1453)
43  org.gnu.Emacs 	0x0007b254 command_loop_2 + 0x40 (keyboard.c:1315)
44  org.gnu.Emacs 	0x000e3ee0 internal_catch + 0xf8 (eval.c:1211)
45  org.gnu.Emacs 	0x0007b1ac command_loop + 0x90 (keyboard.c:1298)
46  org.gnu.Emacs 	0x0007ab94 recursive_edit_1 + 0xa8 (keyboard.c:988)
47  org.gnu.Emacs 	0x0007ad2c Frecursive_edit + 0xdc (keyboard.c:1049)
48  org.gnu.Emacs 	0x0007982c main + 0xc38 (emacs.c:1783)
49  org.gnu.Emacs 	0x0000a010 _start + 0x188 (crt.c:267)
50  org.gnu.Emacs 	0x00009e84 start + 0x30

>>> So I suspect that some part of Emacs' buffer and memory management
>>> fails to check the result of malloc calls or something similar.

>EZ> I'd rather suspect something else: that some library function on your
>EZ> platform is passed a pointer to buffer text, and that library function
>EZ> calls malloc internally, which (especially when Emacs needs to
>EZ> allocate a large amount of memory) causes Emacs to relocate buffer
>EZ> text, which could easily invalidate the pointer(s) used by that
>EZ> library function.

But then it should occur at random times. And my observation is that
in the 2 or 3 cases that this happened the systems disk was
full, and/or the system warned about a lack of swap space.

>>> I have written a test program and it appears that Mac OS X's malloc
>>> and realloc correctly return NULL when the process runs out of
>>> memory.

>EZ> Does Emacs indeed use system malloc on your platform?  Or does it use
>EZ> gmalloc?

darwin.h does define SYSTEM_MALLOC.
And the configure says:
  Should Emacs use the GNU version of malloc?             no
    (The GNU allocators don't work with this system configuration.)
  Should Emacs use a relocating allocator for buffers?    no
  Should Emacs use mmap(2) for buffer allocation?         no

The kernel gives an error around the time that the buffers got mixed
up:
no space in available paging segments
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: Possible memory corruption problem

[Richard M. Stallman]

    I'd rather suspect something else: that some library function on your
    platform is passed a pointer to buffer text, and that library function
    calls malloc internally, which (especially when Emacs needs to
    allocate a large amount of memory) causes Emacs to relocate buffer
    text, which could easily invalidate the pointer(s) used by that
    library function.

It is very rare in Emacs to pass a library function a pointer to
buffer text.  And in most cases these library functions are low-level,
such as memcpy, read, or write.  They should not call malloc.

Is there any place where Emacs calls other C library functions
on buffer text?

    > I have tried to find the common circumstances when this happens and it
    > aeems to me that it happens when the machine is low on virtual memory
    > (including swap space).

    Emacs should display a warning when the system is low on memory.  Does
    it?

Emacs tries to estimate how much memory is available, but that estimate
may not really work.  For instance, it never works for me.
The code to estimate available space worked in the 80s on Unix,
but it may need adaptation to the systems of today.

Re: Possible memory corruption problem

[Eli Zaretskii]

> From: "Richard M. Stallman" <rms@gnu.org>
> CC: piet@cs.uu.nl, emacs-devel@gnu.org
> Date: Sun, 12 Feb 2006 23:40:51 -0500
> 
>     > I have tried to find the common circumstances when this happens and it
>     > aeems to me that it happens when the machine is low on virtual memory
>     > (including swap space).
> 
>     Emacs should display a warning when the system is low on memory.  Does
>     it?
> 
> Emacs tries to estimate how much memory is available, but that estimate
> may not really work.  For instance, it never works for me.
> The code to estimate available space worked in the 80s on Unix,
> but it may need adaptation to the systems of today.

It's possible that the existing estimate doesn't work on systems that
don't use sbrk.  I believe GNU/Linux is one of those systems.

Re: Possible memory corruption problem

[Piet van Oostrum]

>>>>> Eli Zaretskii <eliz@gnu.org> (EZ) wrote:

>>> From: "Richard M. Stallman" <rms@gnu.org>
>>> CC: piet@cs.uu.nl, emacs-devel@gnu.org
>>> Date: Sun, 12 Feb 2006 23:40:51 -0500
>>> 
>>> > I have tried to find the common circumstances when this happens and it
>>> > aeems to me that it happens when the machine is low on virtual memory
>>> > (including swap space).
>>> 
>>> Emacs should display a warning when the system is low on memory.  Does
>>> it?
>>> 
>>> Emacs tries to estimate how much memory is available, but that estimate
>>> may not really work.  For instance, it never works for me.
>>> The code to estimate available space worked in the 80s on Unix,
>>> but it may need adaptation to the systems of today.

>EZ> It's possible that the existing estimate doesn't work on systems that
>EZ> don't use sbrk.  I believe GNU/Linux is one of those systems.

It seems it is not used on Mac OS X either. The man page says:
     The brk and sbrk functions are historical curiosities left over from ear-
     lier days before the advent of virtual memory management.

And I run a test program with malloc calls, set a breakpoint at brk and
sbrk and it did not reach those.
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: Possible memory corruption problem

[Aidan Kehoe]

 Ar an ceathrú lá déag de mí Feabhra, scríobh Piet van Oostrum: 

 > >>> Emacs tries to estimate how much memory is available, but that estimate
 > >>> may not really work.  For instance, it never works for me.
 > >>> The code to estimate available space worked in the 80s on Unix,
 > >>> but it may need adaptation to the systems of today.

Ben Wing implemented this in XEmacs by looking at what getrlimit(2) returns,
which is effective and consistent with what modern OSes do. He also
abstracted it to fall back to ulimit if that’s not available, or to use the
Win32-specific procedure when that’s appropriate. 

 > >EZ> It's possible that the existing estimate doesn't work on systems that
 > >EZ> don't use sbrk.  I believe GNU/Linux is one of those systems.
 > 
 > It seems it is not used on Mac OS X either. The man page says:
 >      The brk and sbrk functions are historical curiosities left over from
 >      ear- lier days before the advent of virtual memory management.
 > 
 > And I run a test program with malloc calls, set a breakpoint at brk and
 > sbrk and it did not reach those.

Which is well and good; implementing a heap using mmap, for example, means
you can release memory back to the OS once it’s been freed. 

-- 
I AM IN JAIL AND ALLOWED SEND ONLY ONE CABLE SINCE WAS ARRESTED WHILE
MEASURING FIFTEEN FOOT WALL OUTSIDE PALACE AND HAVE JUST FINISHED COUNTING
THIRTY EIGHT THOUSAND FIVE HUNDERED TWENTY TWO NAMES WHOS WHO IN MIDEAST.

Re: Possible memory corruption problem

[Richard M. Stallman]

    It's possible that the existing estimate doesn't work on systems that
    don't use sbrk.  I believe GNU/Linux is one of those systems.

Does anyone know how to estimate the amount of space really available
on a GNU/Linux system?

Re: Possible memory corruption problem

[Richard M. Stallman]

Does this change do the job?  (Of course, we would put HAVE_GETRLIMIT
under the control of configure.)

*** vm-limit.c	07 Feb 2006 18:13:40 -0500	1.16
--- vm-limit.c	14 Feb 2006 22:57:45 -0500	
***************
*** 32,37 ****
--- 32,40 ----
  #endif
  
  #include "mem-limits.h"
+ #include <sys/resource.h>
+ 
+ #define HAVE_GETRLIMIT
  
  /*
    Level number of warnings already issued.
***************
*** 61,66 ****
--- 64,82 ----
    unsigned long five_percent;
    unsigned long data_size;
  
+ #ifdef HAVE_GETRLIMIT
+   struct rlimit {
+     rlim_t rlim_cur;
+     rlim_t rlim_max;
+   } rlimit;
+ 
+   getrlimit (RLIMIT_DATA, &rlimit);
+ 
+   five_percent = rlimit.rlim_max / 20;
+   data_size = rlimit.rlim_cur;
+ 
+ #else /* not HAVE_GETRLIMIT */
+ 
    if (lim_data == 0)
      get_lim_data ();
    five_percent = lim_data / 20;
***************
*** 73,78 ****
--- 89,96 ----
  #endif
    cp = (char *) (*__morecore) (0);
    data_size = (char *) cp - (char *) data_space_start;
+ 
+ #endif /* not HAVE_GETRLIMIT */
  
    if (warn_function)
      switch (warnlevel)

Re: Possible memory corruption problem

[Piet van Oostrum]

>>>>> "Richard M. Stallman" <rms@gnu.org> (RMS) wrote:

>RMS> Does this change do the job?  (Of course, we would put HAVE_GETRLIMIT
>RMS> under the control of configure.)

I think under normal circumstances this would generate the warnings.
However, man getrlimit in Mac OS X says:


     RLIMIT_DATA     The maximum size (in bytes) of the data segment for a
                     process; this defines how far a program may extend its
                     break with the sbrk(2) system call.

I don't know if this is to be taken literally, because as I wrote in a
previous message sbrk() isn't used at all.
Also I doubt that this will give a warning if the memory shortage is caused
by depletion of swap space rather than running out of address space.

Moreover I think this patch does not solve the problem at hand, as these
are only warnings. My assessment of the situation was that Emacs got an
unsuccessful malloc or realloc and didn't notice it. When Emacs gets NULL
on these it should abort or some such whether there has been a low memory
warning or not, and apparently it just continued.

-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: Possible memory corruption problem

[Richard M. Stallman]

    My assessment of the situation was that Emacs got an
    unsuccessful malloc or realloc and didn't notice it. When Emacs gets NULL
    on these it should abort or some such whether there has been a low memory
    warning or not, and apparently it just continued.

Emacs normally calls xmalloc, which tests for failure.  But in that
case it calls plain malloc.  I changed it to call xmalloc in that
case.

Does that fix the problem?