Re: VM crashes on big INBOX file ==> problem found!!!

Re: VM crashes on big INBOX file ==> problem found!!!

[Baoqiu.Cui]

Baoqiu.Cui@yahoo.com writes:

> Reiner Steib <reinersteib+from-uce@imap.cc> writes:
>
>>>>I also tried splitting this big folder into two (one with 44MB and
>>>>one with 35MB), and VM can also crash on the small ones sometimes.
>>
>> Please clarify if *Emacs* or *VM* crashes.  (What to you mean by
>> "crash"?)
>
> It is Emacs that crashes...  Everything is gone except a line of
> "Segmentation fault    emacs" on the console.  I can easily recreate
> this using the 79MB folder.

OK.  After running GDB on Emacs, I finally found the real cause for the
crash.  The problem is that, when VM is visiting the huge 79MB folder,
garbage collection is triggered and function mark_object() (in alloc.c)
is recursively called about 29,885 times (see the backtrace info
below)!!!  So many levels of mark_object() calls make the stack
overflow, causing a segmentation fault.

Since I am running Emacs on Cygwin, I had to change the following line
in src/Makefile

  LD=$(CC) -Wl,--image-base,0x20000000

to

  LD=$(CC) -Wl,--image-base,0x20000000,--stack,8000000

to increase the stack limit to ~8MB and have the problem resolved.
After making this change and rebuilding Emacs, now I am able to read the
huge folder without any problem.

I hope Emacs can do a better job in adjusting the stack limit, or at
least add some document in etc/PROBLEMS.

Here is the backtrace information before Emacs crashes:
------------------------------------------------------------------------
Current directory is ~/tmp/emacs-22.1/src/
GNU gdb 6.5.50.20060706-cvs (cygwin-special)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-cygwin"...
DISPLAY = localhost:0.0
TERM = dumb
Breakpoint 1 at 0x2009ff96: file emacs.c, line 431.
Breakpoint 2 at 0x200b9ed9: file sysdep.c, line 1385.
(gdb) r
Starting program: /cygdrive/c/tmp/emacs-22.1/src/emacs.exe -geometry 80x40+0+0
Loaded symbols for /cygdrive/c/WINDOWS/system32/ntdll.dll
Loaded symbols for /cygdrive/c/WINDOWS/system32/kernel32.dll
Loaded symbols for /usr/bin/cygncurses-8.dll
Loaded symbols for /usr/bin/cygwin1.dll
Loaded symbols for /cygdrive/c/WINDOWS/system32/advapi32.dll
Loaded symbols for /cygdrive/c/WINDOWS/system32/rpcrt4.dll
Loaded symbols for /usr/bin/cygjpeg-62.dll
Loaded symbols for /usr/bin/cygpng12.dll
Loaded symbols for /usr/bin/cygz.dll
Loaded symbols for /usr/bin/cygtiff-5.dll
Loaded symbols for /usr/bin/cygungif-4.dll
Loaded symbols for /usr/X11R6/bin/cygX11-6.dll
Loaded symbols for /usr/X11R6/bin/cygICE-6.dll
Loaded symbols for /usr/X11R6/bin/cygSM-6.dll
Loaded symbols for /usr/X11R6/bin/cygXaw3d-7.dll
Loaded symbols for /usr/X11R6/bin/cygXext-6.dll
Loaded symbols for /usr/X11R6/bin/cygXmu-6.dll
Loaded symbols for /usr/X11R6/bin/cygXt-6.dll
Loaded symbols for /usr/X11R6/bin/cygXpm-4.dll

Breakpoint 3 at 0x20080c86: file xterm.c, line 7861.

Program received signal SIGSEGV, Segmentation fault.
mark_object (arg=567788100) at alloc.c:5700
5700		    mark_object (ptr->contents[i]);
(gdb) where
#0  mark_object (arg=567788100) at alloc.c:5700
#1  0x200f4c9d in mark_object (arg=567788132) at alloc.c:5700
#2  0x200f4707 in mark_object (arg=564785749) at alloc.c:5825
#3  0x200f4883 in mark_object (arg=567803073) at alloc.c:5712
#4  0x200f4c9d in mark_object (arg=567807364) at alloc.c:5700
#5  0x200f4c9d in mark_object (arg=567788036) at alloc.c:5700
#6  0x200f4707 in mark_object (arg=564785805) at alloc.c:5825
#7  0x200f4883 in mark_object (arg=567803145) at alloc.c:5712
#8  0x200f4c9d in mark_object (arg=567807236) at alloc.c:5700
#9  0x200f4c9d in mark_object (arg=567787940) at alloc.c:5700
#10 0x200f4707 in mark_object (arg=564785861) at alloc.c:5825
#11 0x200f4883 in mark_object (arg=567803217) at alloc.c:5712
#12 0x200f4c9d in mark_object (arg=567807108) at alloc.c:5700
#13 0x200f4c9d in mark_object (arg=567787844) at alloc.c:5700
#14 0x200f4707 in mark_object (arg=564785917) at alloc.c:5825
#15 0x200f4883 in mark_object (arg=567803289) at alloc.c:5712
#16 0x200f4c9d in mark_object (arg=567806980) at alloc.c:5700
#17 0x200f4c9d in mark_object (arg=567787748) at alloc.c:5700
#18 0x200f4707 in mark_object (arg=564785973) at alloc.c:5825
#19 0x200f4883 in mark_object (arg=567803361) at alloc.c:5712
#20 0x200f4c9d in mark_object (arg=567806852) at alloc.c:5700
#21 0x200f4c9d in mark_object (arg=567787652) at alloc.c:5700

...

#29884 0x200f4707 in mark_object (arg=560521157) at alloc.c:5825
#29885 0x200f4899 in mark_object (arg=565795341) at alloc.c:5714
#29886 0x20145e0d in traverse_intervals_noorder (tree=0x21a518c4, 
    function=0x200f4d80 <mark_interval>, arg=539875329) at intervals.c:218
#29887 0x20145e36 in traverse_intervals_noorder (tree=0x21a518e0, 
    function=0x200f4d80 <mark_interval>, arg=539875329) at intervals.c:223
#29888 0x20145e36 in traverse_intervals_noorder (tree=0x21b9c8c4, 
    function=0x200f4d80 <mark_interval>, arg=539875329) at intervals.c:223
#29889 0x20145e36 in traverse_intervals_noorder (tree=0x21b9d380, 
    function=0x200f4d80 <mark_interval>, arg=539875329) at intervals.c:223
#29890 0x20145e36 in traverse_intervals_noorder (tree=0x21ba0588, 
    function=0x200f4d80 <mark_interval>, arg=539875329) at intervals.c:223
#29891 0x20145e36 in traverse_intervals_noorder (tree=0x21bbf534, 
    function=0x200f4d80 <mark_interval>, arg=539875329) at intervals.c:223
#29892 0x20145e36 in traverse_intervals_noorder (tree=0x21bce4fc, 
    function=0x200f4d80 <mark_interval>, arg=539875329) at intervals.c:223
Debugger finished
------------------------------------------------------------------------

Thanks to all of you for the help!

Regards,
Baoqiu

Re: VM crashes on big INBOX file ==> problem found!!!

[Richard Stallman]

    garbage collection is triggered and function mark_object() (in alloc.c)
    is recursively called about 29,885 times (see the backtrace info
    below)!!!  So many levels of mark_object() calls make the stack
    overflow, causing a segmentation fault.

I wonder if this indicates a bad choice of data structures.
mark_object calls itself recursively for many kinds of pointers,
but it is supposed to loop rather than recurse for cdr pointers.
This is so that long lists do not cause recursion.
Most data structures don't have a tremendous amount of nesting
in the car direction.

I wonder what sort of data structures made 29,885 recursive
calls necessary.  Perhaps we should change those data structures
or else change the garbage collector so it recurses less.

For instance, maybe a lot of this recursion goes thru symbols.  If so,
here is an idea.  Suppose that when mark_object finds a symbol which
is not yet marked, and is interned in the main obarray, it sets a bit
"needs to be marked" in the symbol.  Then increment a counter
which records how many symbols are in this state.

Then gc_sweep could end by scanning the main obarray over and over,
marking those symbols, until the counter goes to zero.

Depending on the nature of the problem, this might or might not
help much.  If the main data types involved are others,
a different solution might help.

Re: VM crashes on big INBOX file ==> problem found!!!

[Baoqiu Cui]

Sorry for the late reply.  I wanted to look deeper into the VM code but
could not find time to do that.  I guess it might be easier for the new
maintainer of VM, Robert Widhopf-Fenk, to take a look at this problem
and see if there is anything that can be changed in VM.  I am copying
this email to Robert...

Thanks for looking into this problem, Richard!

- Baoqiu

Richard Stallman writes:
 >     garbage collection is triggered and function mark_object() (in alloc.c)
 >     is recursively called about 29,885 times (see the backtrace info
 >     below)!!!  So many levels of mark_object() calls make the stack
 >     overflow, causing a segmentation fault.
 > 
 > I wonder if this indicates a bad choice of data structures.
 > mark_object calls itself recursively for many kinds of pointers,
 > but it is supposed to loop rather than recurse for cdr pointers.
 > This is so that long lists do not cause recursion.
 > Most data structures don't have a tremendous amount of nesting
 > in the car direction.
 > 
 > I wonder what sort of data structures made 29,885 recursive
 > calls necessary.  Perhaps we should change those data structures
 > or else change the garbage collector so it recurses less.
 > 
 > For instance, maybe a lot of this recursion goes thru symbols.  If so,
 > here is an idea.  Suppose that when mark_object finds a symbol which
 > is not yet marked, and is interned in the main obarray, it sets a bit
 > "needs to be marked" in the symbol.  Then increment a counter
 > which records how many symbols are in this state.
 > 
 > Then gc_sweep could end by scanning the main obarray over and over,
 > marking those symbols, until the counter goes to zero.
 > 
 > Depending on the nature of the problem, this might or might not
 > help much.  If the main data types involved are others,
 > a different solution might help.

Re: VM crashes on big INBOX file ==> problem found!!!

[Richard Stallman]

This problem may be triggered by something about the data structures
made by VM, but the bug is certainly not in VM.  It is in Emacs.  And
the only way to debug it is to investigate with GDB looking at a
crash.

Can you collect sufficient data to make a test case with which someone
else can observe the bug?

Re: VM crashes on big INBOX file

[Baoqiu.Cui]

Baoqiu.Cui@yahoo.com writes:

> Reiner Steib <reinersteib+from-uce@imap.cc> writes:
>
>>>>I also tried splitting this big folder into two (one with 44MB and
>>>>one with 35MB), and VM can also crash on the small ones sometimes.
>>
>> Please clarify if *Emacs* or *VM* crashes.  (What to you mean by
>> "crash"?)
>
> It is Emacs that crashes...  Everything is gone except a line of
> "Segmentation fault    emacs" on the console.  I can easily recreate
> this using the 79MB folder.

OK.  After running GDB on Emacs, I finally found the real cause for the
crash.  The problem is that, when VM is visiting the huge 79MB folder,
garbage collection is triggered and function mark_object() (in alloc.c)
is recursively called about 29,885 times!!!  Too many levels of
mark_object() calls make the stack overflow, causing a segmentation
fault.

Since I am running Emacs on Cygwin, I had to change the following line
in src/Makefile

  LD=$(CC) -Wl,--image-base,0x20000000

to something like

  LD=$(CC) -Wl,--image-base,0x20000000,--stack,8000000

to increase the stack limit to ~8MB.  After making this change and
rebuilding Emacs, now I am able to read the huge folder without any
problem.

Maybe some comments could be added in etc/PROBLEMS for such a problem.
Apparently, XEmacs has a higher stack limit on Cygwin.

Thanks to all of your for the help!

- Baoqiu