Latest guile 1.9 segfault in GC

Latest guile 1.9 segfault in GC

[Cedric Cellier]

% cat main.c 

#include <stdlib.h>
#include <pthread.h>
#include <libguile.h>

pthread_t dumper;

static void *call_gc(void *null)
{
    scm_gc();
    return NULL;
}
    
void *thread_dumper(void *null)
{
    scm_with_guile(call_gc, NULL);
    return NULL;
}

int main(void)
{
    while (1) {
        pthread_create(&dumper, NULL, thread_dumper, NULL);
        for (;;) sleep(1);
    }

    return EXIT_SUCCESS;
}

% make
gcc -O0 -g3 -ggdb3 -I/usr/local/include/guile/2.0 -I/usr/local/include      main.c  -L/usr/local/lib -lguile-2.0 -lgc -lunistring -lcrypt -lm -lltdl   -lpthread -o main
% ./main 
zsh: segmentation fault (core dumped)  ./main
% gdb main core
...
(gdb) bt
#0  0x00007f184630b698 in GC_is_black_listed (h=0x24ea000, len=4096) at blacklst.c:233
#1  0x00007f1846308c15 in GC_allochblk_nth (sz=512, kind=1, flags=0, n=14, may_split=1) at allchblk.c:690
#2  0x00007f1846308989 in GC_allochblk (sz=512, kind=1, flags=0) at allchblk.c:625
#3  0x00007f18463185f2 in GC_new_hblk (gran=32, kind=1) at new_hblk.c:191
#4  0x00007f184630b090 in GC_allocobj (gran=32, kind=1) at alloc.c:1290
#5  0x00007f1846311010 in GC_generic_malloc_inner (lb=496, k=1) at malloc.c:126
#6  0x00007f1846310fd9 in GC_generic_malloc_inner (lb=496, k=1) at malloc.c:121
#7  0x00007f18463111b4 in GC_generic_malloc (lb=496, k=1) at malloc.c:166
#8  0x00007f1846311644 in GC_core_malloc (lb=496) at malloc.c:276
#9  0x00007f184631d19f in GC_malloc (bytes=496) at thread_local_alloc.c:161
#10 0x00007f18465f6d6e in scm_gc_malloc (size=496, what=0x7f18466d8cd0 "thread") at gc-malloc.c:189
#11 0x00007f184666f0b3 in guilify_self_1 (base=0x7f1844af9d88) at threads.c:325
#12 0x00007f1846671a3e in scm_threads_prehistory (base=0x7f1844af9d88) at threads.c:1924
#13 0x00007f184660b9a5 in scm_i_init_guile (base=0x7f1844af9d88) at init.c:442
#14 0x00007f184666f718 in scm_i_init_thread_for_guile (base=0x7f1844af9d88, parent=0x0) at threads.c:582
#15 0x00007f184666f8c7 in scm_i_with_guile_and_parent (func=0x4007f4 <call_gc>, data=0x0, parent=0x0) at threads.c:730
#16 0x00007f184666f866 in scm_with_guile (func=0x4007f4 <call_gc>, data=0x0) at threads.c:713
#17 0x0000000000400827 in thread_dumper (null=0x0) at main.c:15
#18 0x00007f184631e66e in GC_inner_start_routine (sb=0x7f1844af9e80, arg=0x24e8fc0) at pthread_support.c:1216
#19 0x00007f1846317d97 in GC_call_with_stack_base (fn=0x7f184631e572 <GC_inner_start_routine>, arg=0x24e8fc0) at misc.c:1361
#20 0x00007f184631e6df in GC_start_routine (arg=0x24e8fc0) at pthread_support.c:1247
#21 0x00007f18457079ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#22 0x00007f18454646fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#23 0x0000000000000000 in ?? ()

% guile
GNU Guile 1.9.11.259-252f

Which I just compiled from commit 252f9f187ac8608ebcbf513bbbda49fc085f3a52

Any idea welcome.

Re: Latest guile 1.9 segfault in GC

[Cedric Cellier]

I'm using libgc v 7.2alpha4.

Apparently the value of GC_all_interior_pointers is 1 when initializing
the GC (so GC_old_normal_bl is not initialized) and suddenly is set to 1
so that the next call to GC_is_black_listed dereference it and thus
crash.

It's set to 0 by scm_storage_prehistory in liguile.

I do not understand how this GC works, but find it strange that this global
variable is changed like this, especially since the libgc gc.h reads :
"GC_all_interior_pointers (...) May not be changed after GC initialization."

How come the libgc GC_init function was already called when
scm_storage_prehistory is called ? Apparently, the libgc overload
pthread_create with its GC_pthread_create().

So, guile must be inited before starting any thread.

Let's try...

OK, it works.

:-)

Re: Latest guile 1.9 segfault in GC

[Cedric Cellier]

Ok, this bug was fixed but not the actual bug I was after.

But finaly, thanks to libgc's debug message, I got it.
Actually, libgc overwrite pthread_create (and pthread_sigmask,
and a few more) to know when a new thread is started.
But this overloading is done "softly", ie. by a define in a
C header file.

The program I'm working on was composed of many compilation unit,
of which only a few include libguile (which then include gc.h).
Some of these compilation unit were calling pthread_create without
including libguile, and then when these threads were garbage
collecting libgc crashed.

So, in conclusion : always include libguile.h in the compilation
units calling pthread functions.

I would have prefered another solution, but I think we can live with
that, _if_ it's documented somewhere.

As a side note, I find libguile 1.9 slower than libguile 1.8.7.
I suppose, being an unstable version, it's compiled by default
with many debug options on. Is there an easy way to configure it
for fast execution, for benchmarking reason ?

Re: Latest guile 1.9 segfault in GC

[Ludovic Courtès]

Hi!

Cedric Cellier <rixed@happyleptic.org> writes:

> I do not understand how this GC works, but find it strange that this global
> variable is changed like this, especially since the libgc gc.h reads :
> "GC_all_interior_pointers (...) May not be changed after GC initialization."

‘scm_storage_prehistory’ does that in the right order:

  void
  scm_storage_prehistory ()
  {
    GC_all_interior_pointers = 0;
    GC_set_free_space_divisor (scm_getenv_int ("GC_FREE_SPACE_DIVISOR", 3));

    GC_INIT ();

> How come the libgc GC_init function was already called when
> scm_storage_prehistory is called ? Apparently, the libgc overload
> pthread_create with its GC_pthread_create().
>
> So, guile must be inited before starting any thread.
>
> Let's try...
>
> OK, it works.

Cool.  :-)

Thanks,
Ludo

Re: Latest guile 1.9 segfault in GC

[Ludovic Courtès]

Hi,

Cedric Cellier <rixed@happyleptic.org> writes:

> But finaly, thanks to libgc's debug message, I got it.
> Actually, libgc overwrite pthread_create (and pthread_sigmask,
> and a few more) to know when a new thread is started.
> But this overloading is done "softly", ie. by a define in a
> C header file.

[...]

> So, in conclusion : always include libguile.h in the compilation
> units calling pthread functions.

That’s not entirely true, see Section “Thread Support” at
<http://www.hpl.hp.com/personal/Hans_Boehm/gc/gcdescr.html>:

  All  implementations must intercept  thread creation  and a  few other
  thread-specific calls to allow  enumeration of threads and location of
  thread stacks.  This is current  accomplished with # define's  in gc.h
  (really gc_pthread_redirects.h), or  optionally by using ld's function
  call wrapping mechanism under Linux.

Indeed, here’s what I have on GNU/Linux:

  $ objdump -T ~/soft/lib/libgc.so|grep pthread_create
  000000000001ef80 g    DF .text  000000000000028e  Base        GC_pthread_create
  0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_create

What OS do you use?

Thanks,
Ludo’.

Re: Latest guile 1.9 segfault in GC

[Andy Wingo]

Hi Cedric,

On Tue 17 Aug 2010 07:08, Cedric Cellier <rixed@happyleptic.org> writes:

> As a side note, I find libguile 1.9 slower than libguile 1.8.7.
> I suppose, being an unstable version, it's compiled by default
> with many debug options on. Is there an easy way to configure it
> for fast execution, for benchmarking reason ?

Guile 1.9's interpreter is slower than 1.8's interpreter, but 1.9 has a
compiler, and compiled Scheme runs much faster than 1.8.

You're probably not hitting the compiler for some reason. I think that
scm_primitive_load doesn't support autocompilation, where it probably
should. Is that the function you're using to load your Scheme code?

To work around it for now, call `load' from scheme. I know it's ugly but
at least you get the speed that way. Use scm_primitive_eval (scm_list_2
(scm_from_locale_string ("load"), path)).

Andy
-- 
http://wingolog.org/

Re: Latest guile 1.9 segfault in GC

[rixed]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

> Guile 1.9's interpreter is slower than 1.8's interpreter, but 1.9 has a
> compiler, and compiled Scheme runs much faster than 1.8.

"much faster" is very nice to hear !

> You're probably not hitting the compiler for some reason. (...)
> To work around it for now, call `load' from scheme. I know it's ugly but
> at least you get the speed that way. Use scm_primitive_eval (scm_list_2
> (scm_from_locale_string ("load"), path)).

Thank you I will give it a try, but I remember having seen messages on stdout dfrom guile pretending that it was compiling my code, and having seen some bytecode in the cache. But maybe everything was not compiled. I will try and report next monday.

Again, thank you all for your good work and support !

[-- Attachment #2: Type: text/html, Size: 1097 bytes --]

Re: Latest guile 1.9 segfault in GC

[Cedric Cellier]

-[ Wed, Aug 18, 2010 at 09:16:15AM -0700, Andy Wingo ]----
> You're probably not hitting the compiler for some reason. I think that
> scm_primitive_load doesn't support autocompilation, where it probably
> should. Is that the function you're using to load your Scheme code?

Ok, the file I load with scm_primitive_load is not compiled but the files
loaded by it were actually compiled. So I moved the critical functions
from the first one to the others, and now I think all the usefull functions
are compiled.

1.9.11 version still seams a bit slower than 1.8.7, most of the time
being spent in vm_debug_engine, malloc and free.

> To work around it for now, call `load' from scheme. I know it's ugly but
> at least you get the speed that way. Use scm_primitive_eval (scm_list_2
> (scm_from_locale_string ("load"), path)).

I tried but then apparently the load paths were affected in some way,
since the files loaded from the first evaluated file cannot be found.

Re: Latest guile 1.9 segfault in GC

[Ludovic Courtès]

Hi Cédric,

Cedric Cellier <rixed@happyleptic.org> writes:

> 1.9.11 version still seams a bit slower than 1.8.7,

At doing what?  Is there a minimal test case you could post?

> most of the time being spent in vm_debug_engine,

That’s the bytecode interpreter, so it’s expected.  :-)

> malloc and free.

That’s more surprising.  That could happen, e.g., if your code calls a
lot of C functions that take string arguments, which are converted from
Scheme using scm_to_locale_string, which in turn calls malloc.

Besides, which profiling tool did you use?

Thanks,
Ludo’.