Re: Debugging hints wanted

[Roland Orre <roland.orre@neurologic.se>]

[-- Attachment #1: Type: text/plain, Size: 3007 bytes --]

Hi Ludovic, thanks for your reply
On Mon, 2008-06-30 at 21:42 +0200, Ludovic Courtès wrote:
> Hi,
> 
> Roland Orre <roland.orre@neurologic.se> writes:
> 
> > I need hints on how to find occasional segmentation faults
> > and missed GC references. This relates to 64 bit machines.
> 
> Is it x86-64, IA64, or something else?
What I'm trying to get working now is on x86-64 (Opteron) to be
able to run it on a big large memory computer IA64 (Itanium2).

> The Git repository (the future 1.8.6) contains an important bug fix for
> IA64.  I think there were x86-64-related during the 1.8.x series, too.
> Thus, I'd suggest using the latest Guile on these platforms.

That's a good hint. I'll check out the code and see if I can locate
the changes. Problem is that I've considered switching a few years,
but since the array API changed from 1.8 it would imply a major rework,
possibly causing other issues as the old array API is used in 
hundreds of places in my code, and there may be other API changes
as well.

> > My modules have worked perfectly fine on 32 bit machines but
> > on 64 bits I occasionally get something like
> > #<freed cell 0x2...; GC missed a reference> if I run that
> > code fast, which indicates a threading problem (I do not use
> > threads in this case, but seems like guile does). This does
> > not occur if I run guile through gdb. This happens not too often
> > but it seems to be related to string->symbol symbol->string.
> 
> Is it reproducible?

This is not really reproducable. If I execute the lines quick by
loading it as a file then it occurs with about 60 % probability.
If I execute  the lines in that file, line by one, it does not
occur. To come around that I can see that it may be complaining
at e.g. a string->symbol conversion. If I then simply replace
that with the id  i.e. (lambda(x) x) then it doesn't happen
but probably this relates to the big issue below.

> > My bigger problem though is frequently occurring 
> > segmentation faults or otherwise corrupt pointers.
> >
> >  If I then run the code in gdb I can get
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0x2ae316e4f070 (LWP 6699)]
> > 0x00002ae314b9d091 in scm_gc_mark_dependencies (p=0x97c) at
> > gc-mark.c:441
> > 441      if (SCM_GC_MARK_P (ptr))
> > Current language:  auto; currently c
> 
> Likewise, is it reproducible?  Can you show the full backtrace (it
> should show where 0x97c comes from)?

This is fully reproducible when it happens as shown. Most often
I get a segmentation fault like this. I have attached a full
gdb backtrace from this. This can be produced over and over
with only base address differences.

Sometimes I've got a pointer to some internal structure like
pointing to the procedure of a loop in the middle of a list of
numbers for instance, which is kind illogical as that internal
structure should not be freed. 

> Hope this helps,
> Ludovic.
Best regards
Roland

[-- Attachment #2: backtrace-full.txt --]
[-- Type: text/plain, Size: 10800 bytes --]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x2b966026a070 (LWP 25032)]
0x00002b965dfb8080 in scm_gc_mark_dependencies (p=0x97c) at gc-mark.c:441
441      if (SCM_GC_MARK_P (ptr))
(gdb) bt 30
#0  0x00002b965dfb8080 in scm_gc_mark_dependencies (p=0x97c) at gc-mark.c:441
#1  0x00002b965dfb800f in scm_gc_mark (ptr=0x97c) at gc-mark.c:137
#2  0x00002b965dfb844c in scm_mark_locations (x=0x7fff4d05ef68, n=1690)
    at gc-mark.c:467
#3  0x00002b965dfeaf1b in scm_threads_mark_stacks () at threads.c:996
#4  0x00002b965dfb7f15 in scm_mark_all () at gc-mark.c:94
#5  0x00002b965dfb75dc in scm_igc (what=0x2b965e129a00 "\001") at gc.c:571
#6  0x00002b965dfb749a in scm_gc_for_newcell (freelist=0x2b965e12cc80, 
    free_cells=0x562250) at gc.c:480
#7  0x00002b965df9d6f2 in scm_acons (key=0x2b965daccf10, value=0x2b9660cc9030, 
    alist=0x2b965dc2fee0) at ../libguile/inline.h:79
#8  0x00002b965dfb3afb in scm_dapply (proc=0x2b965dc2fec0, arg1=0x97c, 
    args=0x2b9660cc9030) at eval.c:4592
#9  0x00002b965dfaf32d in scm_apply (proc=0x2b965dc2fec0, arg1=0x2b965db1b9c0, 
    args=0x2b9660cc9050) at eval.c:4407
#10 0x00002b965dfaeab8 in scm_call_3 (proc=0x2b965dc2fec0, 
    arg1=0x2b965db1b9c0, arg2=0x97c, arg3=0x1) at eval.c:4280
#11 0x00002b965dfc9565 in module_variable (module=0x2b965db1b9c0, 
    sym=0x2b965db22060) at modules.c:291
#12 0x00002b965dfc95a2 in module_variable (module=0x2b965daff8e0, 
    sym=0x2b965db22060) at modules.c:301
#13 0x00002b965dfc9995 in scm_sym2var (sym=0x2b965db22060, proc=0x0, 
    definep=0x1) at modules.c:454
#14 0x00002b965dfb4792 in scm_defined_p (sym=0x2b965db22060, env=0x57c)
    at evalext.c:68
#15 0x00002b965dfb10a4 in scm_deval (x=0x97c, env=0x2b9660cc9900)
    at eval.c:3805
#16 0x00002b965dfb1ab0 in scm_deval (x=0x2b965dafb7d0, env=0x2b9660cc9900)
    at eval.c:3139
#17 0x00002b965dfb1b78 in scm_deval (x=0x2b9660aa2fb0, env=0x2b9660cc9900)
    at eval.c:3181
#18 0x00002b965dfb0e5f in scm_deval (x=0x2b965db75940, env=0x2b9660cc9bf0)
    at eval.c:2962
#19 0x00002b965dfb0e5f in scm_deval (x=0x2b965db752c0, env=0x2b9660cc9cc0)
    at eval.c:2962
#20 0x00002b965dfb1c0d in scm_deval (x=0x2b965dad3e50, env=0x2b9660cc6f50)
    at eval.c:3203
#21 0x00002b965dfb2c52 in scm_deval (x=0x2b965dae5970, env=0x2b9660cc7020)
    at eval.c:3799
#22 0x00002b965dfb0e5f in scm_deval (x=0x2b9660a9d970, env=0x2b9660bafa10)
    at eval.c:2962
#23 0x00002b965dfb0e5f in scm_deval (x=0x2b9660a7fce0, env=0x2b9660bafc70)
    at eval.c:2962
#24 0x00002b965dfb3ba2 in scm_dapply (proc=0x2b9660a88bc0, arg1=0x97c, 
    args=0x2b9660ba85f0) at eval.c:4619
#25 0x00002b965dfaf32d in scm_apply (proc=0x2b9660a88be0, arg1=0x2b965db24600, 
    args=0x2b965da82640) at eval.c:4407
#26 0x00002b965dfaea24 in scm_call_1 (proc=0x97c, arg1=0x1) at eval.c:4268
#27 0x00002b965dfafc1d in scm_map (proc=0x2b9660a88be0, arg1=0x2b965dbec760, 
    args=0x97c) at eval.c:5120
#28 0x00002b965dfb125f in scm_deval (x=0x97c, env=0x2b9660a96c80)
    at eval.c:3951
#29 0x00002b965dfb1e09 in scm_deval (x=0x2b9660a8e450, env=0x2b9660a96c80)
    at eval.c:3244
#30 0x00002b965dfb0e5f in scm_deval (x=0x2b9660a8e490, env=0x2b9660a96c80)
    at eval.c:2962
#31 0x00002b965dfb0db4 in scm_deval (x=0x2b9660a49c40, env=0x2b9660a4d790)
    at eval.c:2936
#32 0x00002b965dfb0db4 in scm_deval (x=0x2b9660a49df0, env=0x2b9660a4d790)
    at eval.c:2936
#33 0x00002b965dfb0454 in scm_i_eval_x (exp=0x97c, env=0x1) at eval.c:5398
#34 0x00002b965dfb04d5 in scm_primitive_eval_x (exp=0x2b9660a49670)
    at eval.c:5415
#35 0x00002b965dfc7d3c in load (data=0x97c) at load.c:78
#36 0x00002b965dfa540a in scm_internal_dynamic_wind (
    before=0x525218 <scm_i_root_state_key>, inner=0x2b965dfc7d20 <load>, 
    after=0x2b965dfc7cc0 <swap_port>, inner_data=0x2b96602775b0, 
    guard_data=0x7fff4d0605a0) at dynwind.c:144
#37 0x00002b965dfc7dfb in scm_primitive_load (filename=0x2b96602775b0)
    at load.c:107
#38 0x00002b965dfb1344 in scm_deval (x=0x97c, env=0x2b9660277520)
    at eval.c:4095
#39 0x00002b965dfb2c52 in scm_deval (x=0x2b965dbdcf20, env=0x2b9660277520)
    at eval.c:3799
#40 0x00002b965dfb0496 in scm_i_eval (exp=0x97c, env=0x2b9660277470)
    at eval.c:5405
#41 0x00002b965dfb0515 in scm_primitive_eval (exp=0x2b9660273710)
    at eval.c:5429
#42 0x00002b965dfb1344 in scm_deval (x=0x97c, env=0x2b9660272fd0)
    at eval.c:4095
#43 0x00002b965dfb0496 in scm_i_eval (exp=0x97c, env=0x2b9660272fd0)
    at eval.c:5405
#44 0x00002b965dfa31ab in scm_start_stack (id=0x5987a0, exp=0x2b965dac7190, 
    env=0x2b9660272fd0) at debug.c:455
#45 0x00002b965dfa323c in scm_m_start_stack (exp=0x2b965dac71a0, 
    env=0x2b9660272fd0) at debug.c:471
#46 0x00002b965dfb35f6 in scm_dapply (proc=0x2b965da865c0, 
    arg1=0x2b965dac71d0, args=0x2b9660272fd0) at eval.c:4480
#47 0x00002b965dfb318e in scm_deval (x=0x2b965dac71d0, env=0x2b9660272fd0)
    at eval.c:3645
#48 0x00002b965dfb1c0d in scm_deval (x=0x2b96602773e0, env=0x2b9660272fd0)
    at eval.c:3203
#49 0x00002b965dfb3ba2 in scm_dapply (proc=0x2b966026fc80, arg1=0x97c, 
    args=0x2b966026fd80) at eval.c:4619
#50 0x00002b965dfb2546 in scm_deval (x=0x2b965da88470, env=0x2b966026fd40)
    at eval.c:3558
#51 0x00002b965dfb2c52 in scm_deval (x=0x2b965daa4670, env=0x2b966026fa90)
    at eval.c:3799
#52 0x00002b965dfb3c08 in scm_dapply (proc=0x2b965daa2080, 
    arg1=0x2b965daa15b0, args=0x2b966026fa90) at eval.c:4615
#53 0x00002b965dfaf32d in scm_apply (proc=0x2b966026fa40, arg1=0x97c, 
    args=0x97c) at eval.c:4407
#54 0x00002b965dfaea03 in scm_call_0 (proc=0x97c) at eval.c:4262
#55 0x00002b965dfa2b59 in with_traps_inner (data=0x97c) at debug.c:91
#56 0x00002b965dfa540a in scm_internal_dynamic_wind (
    before=0x525218 <scm_i_root_state_key>, 
    inner=0x2b965dfa2b50 <with_traps_inner>, 
    after=0x2b965dfa2b40 <with_traps_after>, inner_data=0x2b966026fa40, 
    guard_data=0x7fff4d0611bc) at dynwind.c:144
#57 0x00002b965dfa2b9b in scm_with_traps (thunk=0x2b966026fa40) at debug.c:101
#58 0x00002b965dfb1344 in scm_deval (x=0x97c, env=0x2b966026f9f0)
    at eval.c:4095
#59 0x00002b965dfb3ba2 in scm_dapply (proc=0x2b966026f980, arg1=0x97c, 
    args=0x2b966026f9f0) at eval.c:4619
#60 0x00002b965dfaf32d in scm_apply (proc=0x2b966026f9a0, arg1=0x97c, 
    args=0x97c) at eval.c:4407
#61 0x00002b965dfaea03 in scm_call_0 (proc=0x97c) at eval.c:4262
#62 0x00002b965dfa540a in scm_internal_dynamic_wind (
    before=0x525218 <scm_i_root_state_key>, inner=0x2b965dfae9f0 <scm_call_0>, 
    after=0x2b965df9e510 <increase_block>, inner_data=0x2b966026f9a0, 
    guard_data=0x0) at dynwind.c:144
#63 0x00002b965df9e5fc in scm_call_with_unblocked_asyncs (proc=0x2b966026f9a0)
    at async.c:348
#64 0x00002b965dfb1344 in scm_deval (x=0x97c, env=0x2b966026f950)
    at eval.c:4095
#65 0x00002b965dfb3ba2 in scm_dapply (proc=0x2b966026f840, arg1=0x97c, 
    args=0x2b966026f950) at eval.c:4619
#66 0x00002b965dfaf32d in scm_apply (proc=0x2b966026f860, arg1=0x97c, 
    args=0x97c) at eval.c:4407
#67 0x00002b965dfaea03 in scm_call_0 (proc=0x97c) at eval.c:4262
#68 0x00002b965dfec27d in scm_body_thunk (body_data=0x97c) at throw.c:316
#69 0x00002b965dfec160 in scm_internal_lazy_catch (tag=0x37c, 
    body=0x2b965dfec270 <scm_body_thunk>, body_data=0x7fff4d0616b0, handler=0, 
    handler_data=0xa) at throw.c:248
#70 0x00002b965dfec5fb in scm_lazy_catch (key=0x97c, thunk=0x1, 
    handler=0x2b966026f8d0) at throw.c:545
#71 0x00002b965dfb125f in scm_deval (x=0x2b965da9b4d0, env=0x2b966026f800)
    at eval.c:3951
#72 0x00002b965dfb3ba2 in scm_dapply (proc=0x2b966026f700, arg1=0x97c, 
    args=0x2b966026f800) at eval.c:4619
#73 0x00002b965dfaf32d in scm_apply (proc=0x2b966026f720, arg1=0x97c, 
    args=0x97c) at eval.c:4407
#74 0x00002b965dfaea03 in scm_call_0 (proc=0x97c) at eval.c:4262
#75 0x00002b965dfec27d in scm_body_thunk (body_data=0x97c) at throw.c:316
#76 0x00002b965dfebfa2 in scm_internal_catch (tag=0x37c, 
    body=0x2b965dfec270 <scm_body_thunk>, body_data=0x7fff4d061a50, 
    handler=0x2b965dfec290 <scm_handle_by_proc>, handler_data=0x7fff4d061a48)
    at throw.c:172
#77 0x00002b965dfec58b in scm_catch (key=0x97c, thunk=0x1, 
    handler=0x2b966026f790) at throw.c:516
#78 0x00002b965dfb125f in scm_deval (x=0x2b965dab09a0, env=0x2b966026f670)
    at eval.c:3951
#79 0x00002b965dfb1c0d in scm_deval (x=0x2b966026f6b0, env=0x2b966026f670)
    at eval.c:3203
#80 0x00002b965dfb3ba2 in scm_dapply (proc=0x2b966026f580, arg1=0x97c, 
    args=0x2b966026f5f0) at eval.c:4619
#81 0x00002b965dfaf32d in scm_apply (proc=0x2b966026f5a0, arg1=0x97c, 
    args=0x97c) at eval.c:4407
#82 0x00002b965dfaea03 in scm_call_0 (proc=0x97c) at eval.c:4262
#83 0x00002b965dfa540a in scm_internal_dynamic_wind (
    before=0x525218 <scm_i_root_state_key>, inner=0x2b965dfae9f0 <scm_call_0>, 
    after=0x2b965df9e530 <decrease_block>, inner_data=0x2b966026f5a0, 
    guard_data=0x0) at dynwind.c:144
#84 0x00002b965df9e584 in scm_call_with_blocked_asyncs (proc=0x97c)
    at async.c:322
#85 0x00002b965dfb1344 in scm_deval (x=0x97c, env=0x2b966026f410)
    at eval.c:4095
#86 0x00002b965dfb1c0d in scm_deval (x=0x2b966026f130, env=0x2b9660270d00)
    at eval.c:3203
#87 0x00002b965dfb1c0d in scm_deval (x=0x2b9660270b10, env=0x2b9660270ad0)
    at eval.c:3203
#88 0x00002b965dfb3ba2 in scm_dapply (proc=0x2b96602700b0, arg1=0x97c, 
    args=0x2b9660270ad0) at eval.c:4619
#89 0x00002b965dfaf32d in scm_apply (proc=0x2b96602700d0, arg1=0x97c, 
    args=0x97c) at eval.c:4407
#90 0x00002b965dfaea03 in scm_call_0 (proc=0x97c) at eval.c:4262
#91 0x00002b965dfa52a3 in scm_dynamic_wind (in_guard=0x2b9660270080, 
    thunk=0x2b96602700d0, out_guard=0x2b9660270120) at dynwind.c:104
#92 0x00002b965dfad160 in scm_ceval (x=0x2b965daf5980, env=0x2b9660270030)
    at eval.c:3951
#93 0x00002b965dfb0454 in scm_i_eval_x (exp=0x97c, env=0x1) at eval.c:5398
#94 0x00002b965dfb04d5 in scm_primitive_eval_x (exp=0x2b965dace3b0)
    at eval.c:5415
#95 0x00002b965dfb05b9 in inner_eval_x (data=0x97c) at eval.c:5462
#96 0x00002b965dfa540a in scm_internal_dynamic_wind (
    before=0x525218 <scm_i_root_state_key>, 
    inner=0x2b965dfb05b0 <inner_eval_x>, 
    after=0x2b965dfb0570 <restore_environment>, inner_data=0x2b965dace3b0, 
    guard_data=0x2b965dace3a0) at dynwind.c:144

#97 0x00002b965dfb0621 in scm_eval_x (exp=0x2b965dace3b0, module=0x0)
    at eval.c:5471
#98 0x00002b965dfe077e in scm_shell (argc=1, argv=0x7fff4d062558)
    at script.c:672
#99 0x00002b965dfc5bd4 in invoke_main_func (body_data=0x0) at init.c:600
#100 0x00002b965dfc5b87 in scm_boot_guile_1 (base=0x97c, 
    closure=0x7fff4d062440) at init.c:580
#101 0x00002b965dfc5896 in scm_boot_guile (argc=2428, argv=0x1, main_func=0, 
    closure=0x0) at init.c:392
#102 0x0000000000403d90 in main (argc=2428, argv=0x1) at bcpnnx.cc:47