FreeBSD/amd64: bootstrap & memory allocation troubles

FreeBSD/amd64: bootstrap & memory allocation troubles

[Giorgos Keramidas]

Hi all,

I apologize in advance if this reaches you multiple times.  I
posted it earlier today, but haven't received any notification
that it reached the list or bounced, hence the repost.

    Date: Fri, 13 Jan 2006 17:52:27 +0200
    From: Giorgos Keramidas <keramida@ceid.upatras.gr>
    Subject: bootstrap & memory allocation troubles
    To: emacs-devel@gnu.org

Hi all,

Recently, the malloc() implementation in FreeBSD changed to a new
implementation by Jason Evans, and a posix_memalign() implementation
was added to the standard library of the system.

This seems to cause bootstrap problems for cvs-emacs now, but I'm not
sure if this is something we should fix in Emacs or in the libc of
FreeBSD.  The gdb trace that shows the problem is listed below, but
I'd be grateful for any hints related to fixing this.

What is surprising is that the Lisp_String allocated internally by
allocate_string_data is supposed to have a size of 50 bytes, but
trying to access any byte after its 40'th character results in
``inaccessible memory'' faults in gdb (and is probably the reason why
bootstrap fails).

Any ideas how the current posix_memalign() implementation could affect
this, and/or tips to debug it?

----- gdb output ------------------------------------------------------

keramida@flame:/home/keramida/ws/cvs-emacs/emacs$ gdb src/bootstrap-emacs lisp/bootstrap-emacs.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `bootstrap-emacs'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libutil.so.5...done.
Loaded symbols for /lib/libutil.so.5
Reading symbols from /lib/libncurses.so.6...done.
Loaded symbols for /lib/libncurses.so.6
Reading symbols from /lib/libm.so.4...done.
Loaded symbols for /lib/libm.so.4
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  allocate_string_data (s=0x179d380, nchars=50, nbytes=50) at alloc.c:1981
1981      s->data[nbytes] = '\0';
(gdb) bt
#0  allocate_string_data (s=0x179d380, nchars=50, nbytes=50) at alloc.c:1981
#1  0x00000000004abc40 in make_uninit_multibyte_string (nchars=50, nbytes=50) at alloc.c:2448
#2  0x00000000004abd7d in make_specified_string (contents=0x7fffffff9380 "/home/keramida/ws/cvs-emacs/emacs/lisp/loaddefs.el",
    nchars=0, nbytes=50, multibyte=0) at alloc.c:2401
#3  0x0000000000486afd in Fexpand_file_name (name=140737488327552, default_directory=25813859) at fileio.c:1709
#4  0x000000000047bf45 in unlock_file (fn=0) at filelock.c:654
#5  0x000000000047c065 in unlock_all_files () at filelock.c:674
#6  0x0000000000459e5b in shut_down_emacs (sig=11, no_x=32088016, stuff=9132033) at emacs.c:2137
#7  0x000000000045a0aa in fatal_error_signal (sig=11) at emacs.c:413
#8  <signal handler called>
#9  allocate_string_data (s=0x179dc00, nchars=102, nbytes=102) at alloc.c:1981
#10 0x00000000004abc40 in make_uninit_multibyte_string (nchars=102, nbytes=102) at alloc.c:2448
#11 0x00000000004abcdb in make_uninit_string (length=0) at alloc.c:2428
#12 0x00000000004ba46e in make_buffer_string_both (start=464607, start_byte=464607, end=464709, end_byte=464709, props=1)
    at editfns.c:2310
#13 0x00000000004ba5d9 in Fbuffer_substring (start=3716856, end=3717672) at editfns.c:2381
#14 0x00000000004c22c4 in Feval (form=1) at eval.c:2222
#15 0x00000000004c241d in Fsetq (args=0) at eval.c:534
#16 0x00000000004c238d in Feval (form=8590072) at eval.c:2166
#17 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#18 0x00000000004c4a64 in Flet (args=26267557) at eval.c:1053
#19 0x00000000004c238d in Feval (form=8589496) at eval.c:2166
#20 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#21 0x00000000004c238d in Feval (form=8590216) at eval.c:2166
#22 0x00000000004c34d5 in Funwind_protect (args=28122293) at eval.c:1310
#23 0x00000000004c238d in Feval (form=8589256) at eval.c:2166
#24 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#25 0x00000000004c4a64 in Flet (args=28122245) at eval.c:1053
#26 0x00000000004c238d in Feval (form=8589496) at eval.c:2166
#27 0x00000000004c20ac in Feval (form=0) at eval.c:2275
#28 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#29 0x00000000004c28b0 in funcall_lambda (fun=26266400, nargs=0, arg_vector=0x7fffffffa520) at eval.c:3059
#30 0x00000000004c29ea in apply_lambda (fun=26266405, args=9132033, eval_flag=1) at eval.c:2988
#31 0x00000000004c1faa in Feval (form=0) at eval.c:2277
#32 0x00000000004c49ed in Flet (args=26289397) at eval.c:1037
#33 0x00000000004c238d in Feval (form=8589496) at eval.c:2166
#34 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#35 0x00000000004c4848 in Fwhile (args=26287109) at eval.c:1075
#36 0x00000000004c238d in Feval (form=8589448) at eval.c:2166
#37 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#38 0x00000000004bc09d in Fsave_restriction (body=26291077) at editfns.c:3105
#39 0x00000000004c238d in Feval (form=8585088) at eval.c:2166
#40 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#41 0x00000000004b7f3d in Fsave_excursion (args=26286933) at editfns.c:963
#42 0x00000000004c238d in Feval (form=8587536) at eval.c:2166
#43 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#44 0x00000000004b7f3d in Fsave_excursion (args=26290261) at editfns.c:963
#45 0x00000000004c238d in Feval (form=8587536) at eval.c:2166
#46 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#47 0x00000000004c4a64 in Flet (args=26292165) at eval.c:1053
#48 0x00000000004c238d in Feval (form=8589496) at eval.c:2166
#49 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#50 0x00000000004c28b0 in funcall_lambda (fun=26285152, nargs=1, arg_vector=0x7fffffffb210) at eval.c:3059
#51 0x00000000004c29ea in apply_lambda (fun=26285157, args=31225539, eval_flag=1) at eval.c:2988
#52 0x00000000004c1faa in Feval (form=0) at eval.c:2277
#53 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#54 0x00000000004c4c58 in Fcond (args=26281125) at eval.c:409
#55 0x00000000004c238d in Feval (form=8590264) at eval.c:2166
#56 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#57 0x00000000004c4bc3 in FletX (args=26283509) at eval.c:995
#58 0x00000000004c238d in Feval (form=8589544) at eval.c:2166
#59 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#60 0x00000000004c4848 in Fwhile (args=26281269) at eval.c:1075
#61 0x00000000004c238d in Feval (form=8589448) at eval.c:2166
#62 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#63 0x00000000004b7f3d in Fsave_excursion (args=26283157) at editfns.c:963
#64 0x00000000004c238d in Feval (form=8587536) at eval.c:2166
#65 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#66 0x00000000004b7f9d in Fsave_current_buffer (args=26331333) at editfns.c:979
#67 0x00000000004c238d in Feval (form=8587488) at eval.c:2166
#68 0x00000000004c20ac in Feval (form=0) at eval.c:2275
#69 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#70 0x00000000004c4bc3 in FletX (args=26284837) at eval.c:995
#71 0x00000000004c238d in Feval (form=8589544) at eval.c:2166
#72 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#73 0x00000000004c28b0 in funcall_lambda (fun=26280112, nargs=18, arg_vector=0x7fffffffc0a8) at eval.c:3059
#74 0x00000000004c2c85 in Ffuncall (nargs=26280117, args=0x7fffffffc0a0) at eval.c:2934
#75 0x00000000004c4325 in Fapply (nargs=2, args=0x7fffffffc1d0) at eval.c:2371
#76 0x00000000004c22c4 in Feval (form=140737488339408) at eval.c:2222
#77 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#78 0x00000000004c28b0 in funcall_lambda (fun=26279632, nargs=0, arg_vector=0x7fffffffc438) at eval.c:3059
#79 0x00000000004c2c85 in Ffuncall (nargs=26279637, args=0x7fffffffc430) at eval.c:2934
#80 0x00000000004c22c4 in Feval (form=140737488340016) at eval.c:2222
#81 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#82 0x00000000004c4cc8 in Fif (args=23767893) at eval.c:381
#83 0x00000000004c238d in Feval (form=8590312) at eval.c:2166
#84 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#85 0x00000000004c4c58 in Fcond (args=23768021) at eval.c:409
#86 0x00000000004c238d in Feval (form=8590264) at eval.c:2166
#87 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#88 0x00000000004c4bc3 in FletX (args=23769285) at eval.c:995
#89 0x00000000004c238d in Feval (form=8589544) at eval.c:2166
#90 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#91 0x00000000004c4848 in Fwhile (args=23799813) at eval.c:1075
#92 0x00000000004c238d in Feval (form=8589448) at eval.c:2166
#93 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#94 0x00000000004c4a64 in Flet (args=23770741) at eval.c:1053
#95 0x00000000004c238d in Feval (form=8589496) at eval.c:2166
#96 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#97 0x00000000004c238d in Feval (form=8590216) at eval.c:2166
#98 0x00000000004c4ca6 in Fif (args=26246949) at eval.c:380
#99 0x00000000004c238d in Feval (form=8590312) at eval.c:2166
#100 0x00000000004c20ac in Feval (form=0) at eval.c:2275
#101 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#102 0x00000000004c28b0 in funcall_lambda (fun=23799504, nargs=1, arg_vector=0x7fffffffd330) at eval.c:3059
#103 0x00000000004c29ea in apply_lambda (fun=23799509, args=24391845, eval_flag=1) at eval.c:2988
#104 0x00000000004c1faa in Feval (form=0) at eval.c:2277
#105 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#106 0x00000000004c28b0 in funcall_lambda (fun=23707520, nargs=0, arg_vector=0x7fffffffd5d0) at eval.c:3059
#107 0x00000000004c29ea in apply_lambda (fun=23707525, args=9132033, eval_flag=1) at eval.c:2988
#108 0x00000000004c1faa in Feval (form=0) at eval.c:2277
#109 0x00000000004c34d5 in Funwind_protect (args=23659077) at eval.c:1310
#110 0x00000000004c238d in Feval (form=8589256) at eval.c:2166
#111 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#112 0x00000000004c4a64 in Flet (args=23659029) at eval.c:1053
#113 0x00000000004c238d in Feval (form=8589496) at eval.c:2166
#114 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#115 0x00000000004c4cc8 in Fif (args=23662549) at eval.c:381
#116 0x00000000004c238d in Feval (form=8590312) at eval.c:2166
#117 0x00000000004c259f in Fprogn (args=0) at eval.c:432
#118 0x00000000004c28b0 in funcall_lambda (fun=23656192, nargs=0, arg_vector=0x7fffffffdd60) at eval.c:3059
#119 0x00000000004c29ea in apply_lambda (fun=23656197, args=9132033, eval_flag=1) at eval.c:2988
#120 0x00000000004c1faa in Feval (form=0) at eval.c:2277
#121 0x00000000004c0cc2 in internal_condition_case (bfun=0x45ba20 <top_level_2>, handlers=9276305, hfun=0x461000 <cmd_error>)
    at eval.c:1465
#122 0x000000000045ba5a in top_level_1 () at keyboard.c:1338
#123 0x00000000004c0b90 in internal_catch (tag=0, func=0x45ba30 <top_level_1>, arg=9132033) at eval.c:1211
#124 0x000000000045b80b in command_loop () at keyboard.c:1295
#125 0x000000000045b8bc in recursive_edit_1 () at keyboard.c:993
#126 0x000000000045b9d7 in Frecursive_edit () at keyboard.c:1054
#127 0x000000000045aebb in main (argc=28, argv=0x7fffffffe290) at emacs.c:1789
(gdb) p s
$1 = (struct Lisp_String *) 0x179d380
(gdb) p *s
$2 = {size = 50, size_byte = 50, intervals = 0x0, data = 0x1e99fd8 ""}
(gdb) p nbytes
$3 = 50
(gdb) p s->data
$4 = (unsigned char *) 0x1e99fd8 ""
(gdb) p *(s->data)
$5 = 0 '\0'
(gdb) p *(s->data)@20
$6 = '\0' <repeats 19 times>
(gdb) p *(s->data)@50
Cannot access memory at address 0x1e99fd8
(gdb) p *(s->data)@49
Cannot access memory at address 0x1e99fd8
(gdb) p *(s->data)@40
$7 = '\0' <repeats 39 times>
(gdb) p *(s->data)@41
Cannot access memory at address 0x1e99fd8
(gdb) quit
keramida@flame:/home/keramida/ws/cvs-emacs/emacs$

Re: FreeBSD/amd64: bootstrap & memory allocation troubles

[Giorgos Keramidas]

On 2006-01-13 21:18, Giorgos Keramidas <keramida@ceid.upatras.gr> wrote:
> keramida@flame:/home/keramida/ws/cvs-emacs/emacs$ gdb src/bootstrap-emacs lisp/bootstrap-emacs.core
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> Core was generated by `bootstrap-emacs'.
> [...]
> #0  allocate_string_data (s=0x179d380, nchars=50, nbytes=50) at alloc.c:1981
> 1981      s->data[nbytes] = '\0';
> (gdb) bt
> #0  allocate_string_data (s=0x179d380, nchars=50, nbytes=50) at alloc.c:1981
> #1  0x00000000004abc40 in make_uninit_multibyte_string (nchars=50, nbytes=50) at alloc.c:2448
> #2  0x00000000004abd7d in make_specified_string (contents=0x7fffffff9380 "/home/keramida/ws/cvs-emacs/emacs/lisp/loaddefs.el",
>     nchars=0, nbytes=50, multibyte=0) at alloc.c:2401
> #3  0x0000000000486afd in Fexpand_file_name (name=140737488327552, default_directory=25813859) at fileio.c:1709
> [...]
> (gdb) p s
> $1 = (struct Lisp_String *) 0x179d380
> (gdb) p *s
> $2 = {size = 50, size_byte = 50, intervals = 0x0, data = 0x1e99fd8 ""}
> (gdb) p nbytes
> $3 = 50
> (gdb) p s->data
> $4 = (unsigned char *) 0x1e99fd8 ""
> (gdb) p *(s->data)
> $5 = 0 '\0'
> (gdb) p *(s->data)@20
> $6 = '\0' <repeats 19 times>
> (gdb) p *(s->data)@50
> Cannot access memory at address 0x1e99fd8
> (gdb) p *(s->data)@49
> Cannot access memory at address 0x1e99fd8
> (gdb) p *(s->data)@40
> $7 = '\0' <repeats 39 times>
> (gdb) p *(s->data)@41
> Cannot access memory at address 0x1e99fd8
> (gdb) quit
> keramida@flame:/home/keramida/ws/cvs-emacs/emacs$

As an added data-point while I continue tracking this, I fixed a bug in
the irssi IRC client last night that would similarly end up accessing
memory areas that are ``inaccessible''.

The new malloc() in FreeBSD may use mmap() to map allocated regions in
the virtual address space of the process, which some times ends up using
addresses above 4 GB.  These addresses cannot fit in an `int' anymore,
which means that if a function is supposed to return a pointer and it's
called from another source file without access to a proper prototype,
then the default return type of `int' assigned to it by GCC cause a
conversion (char *) -> int -> (char *).  The first conversion to `int'
truncates the high bits of teh address, which *are* significant, causing
addresses like 0x80123456 to end up being returned as 0x123456, which
triggers the core dump.

I think somewhere along the stack trace shown in the original message of
this thread something similar happens, but I'll post again when I find
out where this happens.