Strange deadlocks in guile

Strange deadlocks in guile

[Stanislav Ievlev]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Greetings!

On my 2.6.18 kernel guile (latest snapshot from git, branch-release-1.8
head)  in some situations (unfortunately I can't made a testcase) stops
forever. Guile with 2.6.27 kernel works well.

Strace:
futex(0xc571890, FUTEX_WAIT, 2, NULL...

Backtrace:
--
#0  0x00002b4ba31b4004 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b4ba31af944 in _L_lock_55 () from /lib64/libpthread.so.0
#2  0x00002b4ba31af341 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b4ba22f87dc in scm_i_thread_put_to_sleep () from
/usr/lib64/libguile.so.17
#4  0x00002b4ba22abfd9 in scm_i_gc () from /usr/lib64/libguile.so.17
#5  0x00002b4ba22ac338 in scm_gc_for_newcell () from
/usr/lib64/libguile.so.17
#6  0x00002b4ba22cc86f in scm_cons () from /usr/lib64/libguile.so.17
--

How can I debug and solve this problem?


--
With best results
Stanislav Ievlev.

[-- Attachment #2: Type: text/html, Size: 1057 bytes --]

Re: Strange deadlocks in guile

[Neil Jerram]

Stanislav Ievlev <stanislav.ievlev@gmail.com> writes:

> Greetings!
>
> On my 2.6.18 kernel guile (latest snapshot from git, branch-release-1.8 head) 
> in some situations (unfortunately I can't made a testcase) stops forever. Guile
> with 2.6.27 kernel works well.
>
> Strace:
> futex(0xc571890, FUTEX_WAIT, 2, NULL...
>
> Backtrace:
> --
> #0  0x00002b4ba31b4004 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00002b4ba31af944 in _L_lock_55 () from /lib64/libpthread.so.0
> #2  0x00002b4ba31af341 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00002b4ba22f87dc in scm_i_thread_put_to_sleep () from /usr/lib64/
> libguile.so.17
> #4  0x00002b4ba22abfd9 in scm_i_gc () from /usr/lib64/libguile.so.17
> #5  0x00002b4ba22ac338 in scm_gc_for_newcell () from /usr/lib64/libguile.so.17
> #6  0x00002b4ba22cc86f in scm_cons () from /usr/lib64/libguile.so.17
> --
>
> How can I debug and solve this problem?

The backtrace above only has 7 frames, and there should be many more.
Can you get the full backtrace, and also backtraces of all the other
threads?

We've recently been working quite a bit on deadlocks, so I'd say
there's a good chance we'll be able to understand and fix this.

Thanks,
        Neil

Re: Strange deadlocks in guile

[Stanislav Ievlev]

[-- Attachment #1.1: Type: text/plain, Size: 1424 bytes --]

It's very strange but I see an only one thread. full Backtrace in attach.


2009/3/12 Neil Jerram <neil@ossau.uklinux.net>

> Stanislav Ievlev <stanislav.ievlev@gmail.com> writes:
>
> > Greetings!
> >
> > On my 2.6.18 kernel guile (latest snapshot from git, branch-release-1.8
> head)
> > in some situations (unfortunately I can't made a testcase) stops forever.
> Guile
> > with 2.6.27 kernel works well.
> >
> > Strace:
> > futex(0xc571890, FUTEX_WAIT, 2, NULL...
> >
> > Backtrace:
> > --
> > #0  0x00002b4ba31b4004 in __lll_lock_wait () from /lib64/libpthread.so.0
> > #1  0x00002b4ba31af944 in _L_lock_55 () from /lib64/libpthread.so.0
> > #2  0x00002b4ba31af341 in pthread_mutex_lock () from
> /lib64/libpthread.so.0
> > #3  0x00002b4ba22f87dc in scm_i_thread_put_to_sleep () from /usr/lib64/
> > libguile.so.17
> > #4  0x00002b4ba22abfd9 in scm_i_gc () from /usr/lib64/libguile.so.17
> > #5  0x00002b4ba22ac338 in scm_gc_for_newcell () from
> /usr/lib64/libguile.so.17
> > #6  0x00002b4ba22cc86f in scm_cons () from /usr/lib64/libguile.so.17
> > --
> >
> > How can I debug and solve this problem?
>
> The backtrace above only has 7 frames, and there should be many more.
> Can you get the full backtrace, and also backtraces of all the other
> threads?
>
> We've recently been working quite a bit on deadlocks, so I'd say
> there's a good chance we'll be able to understand and fix this.
>
> Thanks,
>         Neil
>

[-- Attachment #1.2: Type: text/html, Size: 2005 bytes --]

[-- Attachment #2: guile18.bt --]
[-- Type: application/octet-stream, Size: 7241 bytes --]

GNU gdb 6.6-alt3 (ALT Linux)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-alt-linux".
Attaching to process 4698
Reading symbols from /usr/bin/guile18...(no debugging symbols found)...done.
Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libguile.so.17...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libguile.so.17
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libgmp.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgmp.so.3
Reading symbols from /lib64/libcrypt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/libltdl.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libltdl.so.3
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 47935337836272 (LWP 4698)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libguile-vhttpd.so...done.
Loaded symbols for /usr/lib64/libguile-vhttpd.so
Reading symbols from /lib64/libglib-2.0.so.0...done.
Loaded symbols for /lib64/libglib-2.0.so.0
Reading symbols from /usr/lib64/libvhttpd.so.1...done.
Loaded symbols for /usr/lib64/libvhttpd.so.1
Reading symbols from /lib64/libpcre.so.3...done.
Loaded symbols for /lib64/libpcre.so.3
Reading symbols from /lib64/libssl.so.7...done.
Loaded symbols for /lib64/libssl.so.7
Reading symbols from /lib64/libcrypto.so.7...done.
Loaded symbols for /lib64/libcrypto.so.7
Reading symbols from /lib64/libkrb5.so.3...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libk5crypto.so.3...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libkrb5support.so.0...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libguile-srfi-srfi-1-v-3.so...done.
Loaded symbols for /usr/lib64/libguile-srfi-srfi-1-v-3.so
Reading symbols from /usr/lib64/libguile-pipe.so...done.
Loaded symbols for /usr/lib64/libguile-pipe.so
Reading symbols from /lib64/libutil.so.1...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /usr/lib64/libguile-gettext.so...done.
Loaded symbols for /usr/lib64/libguile-gettext.so
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
0x00002b98d0878004 in __lll_lock_wait ()
   from /lib64/libpthread.so.0
(gdb) info threads
  1 Thread 47935337836272 (LWP 4698)  0x00002b98d0878004 in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00002b98d0878004 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b98d0873944 in _L_lock_55 () from /lib64/libpthread.so.0
#2  0x00002b98d0873341 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b98cf9bc7dc in scm_i_thread_put_to_sleep () from /usr/lib64/libguile.so.17
#4  0x00002b98cf96ffd9 in scm_i_gc () from /usr/lib64/libguile.so.17
#5  0x00002b98cf970338 in scm_gc_for_newcell () from /usr/lib64/libguile.so.17
#6  0x00002b98cf981eaf in scm_list_1 () from /usr/lib64/libguile.so.17
#7  0x00002b98cf964099 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#8  0x00002b98cf962d7b in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#9  0x00002b98cf96440f in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#10 0x00002b98cf963fe4 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#11 0x00002b98cf963fe4 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#12 0x00002b98cf96d691 in scm_c_with_fluid () from /usr/lib64/libguile.so.17
#13 0x00002b98cf963359 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#14 0x00002b98cf9bee1f in scm_c_catch () from /usr/lib64/libguile.so.17
#15 0x00002b98cf9bf03e in scm_catch_with_pre_unwind_handler () from /usr/lib64/libguile.so.17
#16 0x00002b98cf962e2a in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#17 0x00002b98cf959f12 in scm_dynamic_wind () from /usr/lib64/libguile.so.17
#18 0x00002b98cf963359 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#19 0x00002b98cf962f03 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#20 0x00002b98cf9642f4 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#21 0x00002b98cf982a22 in scm_primitive_load () from /usr/lib64/libguile.so.17
#22 0x00002b98cf96355b in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#23 0x00002b98cf95627b in scm_start_stack () from /usr/lib64/libguile.so.17
#24 0x00002b98cf962b67 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#25 0x00002b98cf96d691 in scm_c_with_fluid () from /usr/lib64/libguile.so.17
#26 0x00002b98cf963359 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#27 0x00002b98cf959f12 in scm_dynamic_wind () from /usr/lib64/libguile.so.17
#28 0x00002b98cf963359 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#29 0x00002b98cf9639e0 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#30 0x00002b98cf96b44a in scm_eval_x () from /usr/lib64/libguile.so.17
#31 0x00002b98cf99e992 in scm_shell () from /usr/lib64/libguile.so.17
#32 0x00002b98cf97f6df in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#33 0x00002b98cf954cfa in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17
#34 0x00002b98cf9bee1f in scm_c_catch () from /usr/lib64/libguile.so.17
#35 0x00002b98cf9551c1 in scm_i_with_continuation_barrier () from /usr/lib64/libguile.so.17
#36 0x00002b98cf955260 in scm_c_with_continuation_barrier () from /usr/lib64/libguile.so.17
#37 0x00002b98cf9bdcd3 in scm_i_with_guile_and_parent () from /usr/lib64/libguile.so.17
#38 0x00002b98cf97f6a5 in scm_boot_guile () from /usr/lib64/libguile.so.17
#39 0x0000000000400980 in gdb_maybe_valid_type_p ()
#40 0x00002b98cfc307a6 in __libc_start_main () from /lib64/libc.so.6
#41 0x00000000004008a9 in gdb_maybe_valid_type_p ()
#42 0x00007fffdb3ac9b8 in ?? ()
#43 0x000000000000001c in ?? ()
#44 0x0000000000000005 in ?? ()
#45 0x00007fffdb3ad86d in ?? ()
#46 0x00007fffdb3ad87e in ?? ()
#47 0x00007fffdb3ad881 in ?? ()
#48 0x00007fffdb3ad896 in ?? ()
#49 0x00007fffdb3ad899 in ?? ()
#50 0x0000000000000000 in ?? ()
(gdb) q
The program is running.  Quit anyway (and detach it)? (y or n) Detaching from program: /usr/bin/guile18, process 4698

Re: Strange deadlocks in guile

[Ludovic Courtès]

Hi,

Stanislav Ievlev <stanislav.ievlev@gmail.com> writes:

> #6  0x00002b98cf981eaf in scm_list_1 () from /usr/lib64/libguile.so.17
> #7  0x00002b98cf964099 in gdb_maybe_valid_type_p () from /usr/lib64/libguile.so.17

All theses `gdb_*' calls are strange.  They refer to Guile's GDB
interface, which never got to the point of being actually used I think.

How does your distribution modify GDB and/or Guile?

Thanks,
Ludo'.

Re: Strange deadlocks in guile

[Neil Jerram]

Stanislav Ievlev <stanislav.ievlev@gmail.com> writes:

> It's very strange but I see an only one thread. full Backtrace in attach.

That is indeed odd.  Are you using GDB, and the "info threads"
command, and it lists only one thread?

>     > #0  0x00002b4ba31b4004 in __lll_lock_wait () from /lib64/libpthread.so.0
>     > #1  0x00002b4ba31af944 in _L_lock_55 () from /lib64/libpthread.so.0
>     > #2  0x00002b4ba31af341 in pthread_mutex_lock () from /lib64/
>     libpthread.so.0
>     > #3  0x00002b4ba22f87dc in scm_i_thread_put_to_sleep () from /usr/lib64/
>     > libguile.so.17
>     > #4  0x00002b4ba22abfd9 in scm_i_gc () from /usr/lib64/libguile.so.17
>     > #5  0x00002b4ba22ac338 in scm_gc_for_newcell () from /usr/lib64/
>     libguile.so.17
>     > #6  0x00002b4ba22cc86f in scm_cons () from /usr/lib64/libguile.so.17

This backtrace indicates the GC thread trying to lock the "heap mutex"
of all other threads (however many there are), and blocking on one of
those.

(Or possibly the thread_admin_mutex.  Can you get line numbers and
match those against your source code, to see if the relevant line in
scm_i_thread_put_to_sleep is

      scm_i_pthread_mutex_lock (&thread_admin_mutex);

or

	scm_i_pthread_mutex_lock (&t->heap_mutex);

?)

This could happen if your application has another thread that is in
Guile mode but has blocked on something - e.g. select, read, locking
another mutex.  Is that possible?

Regards,
        Neil

Re: Strange deadlocks in guile

[Stanislav Ievlev]

[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]

2009/3/14 Neil Jerram <neil@ossau.uklinux.net>

> Stanislav Ievlev <stanislav.ievlev@gmail.com> writes:
>
> > It's very strange but I see an only one thread. full Backtrace in attach.
>
> That is indeed odd.  Are you using GDB, and the "info threads"
> command, and it lists only one thread?
>
> >     > #0  0x00002b4ba31b4004 in __lll_lock_wait () from
> /lib64/libpthread.so.0
> >     > #1  0x00002b4ba31af944 in _L_lock_55 () from /lib64/libpthread.so.0
> >     > #2  0x00002b4ba31af341 in pthread_mutex_lock () from /lib64/
> >     libpthread.so.0
> >     > #3  0x00002b4ba22f87dc in scm_i_thread_put_to_sleep () from
> /usr/lib64/
> >     > libguile.so.17
> >     > #4  0x00002b4ba22abfd9 in scm_i_gc () from
> /usr/lib64/libguile.so.17
> >     > #5  0x00002b4ba22ac338 in scm_gc_for_newcell () from /usr/lib64/
> >     libguile.so.17
> >     > #6  0x00002b4ba22cc86f in scm_cons () from
> /usr/lib64/libguile.so.17
>
> This backtrace indicates the GC thread trying to lock the "heap mutex"
> of all other threads (however many there are), and blocking on one of
> those.
>
> (Or possibly the thread_admin_mutex.  Can you get line numbers and
> match those against your source code, to see if the relevant line in
> scm_i_thread_put_to_sleep is
>
>      scm_i_pthread_mutex_lock (&thread_admin_mutex);
>
> or
>
>        scm_i_pthread_mutex_lock (&t->heap_mutex);
>
> ?)
>
> This could happen if your application has another thread that is in
> Guile mode but has blocked on something - e.g. select, read, locking
> another mutex.  Is that possible?


Yes, it's possible.

This is the our guile based system configurator (
http://git.altlinux.org/people/inger/packages/alterator.git).
This program calls backends (fork/exec) that written in other languages, and
communicate with them (read/write)  through a simple pipe.

(this is an communitation code:
http://git.altlinux.org/people/inger/packages/alterator.git?p=alterator.git;a=blob;f=alterator/interfaces/guile/ensign/backend3.scm;h=501de2ac19677bf722834522e1098df92a66e7ba;hb=f0744c441a86a85eaced5ad7cb401f318941307e
)

Is it possible to solve it or we'll have to use a single threaded edition of
guile?

[-- Attachment #2: Type: text/html, Size: 3103 bytes --]

Re: Strange deadlocks in guile

[Stanislav Ievlev]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

2009/3/14 Neil Jerram <neil@ossau.uklinux.net>

> Stanislav Ievlev <stanislav.ievlev@gmail.com> writes:
>
> > It's very strange but I see an only one thread. full Backtrace in attach.
>
> That is indeed odd.  Are you using GDB, and the "info threads"
> command, and it lists only one thread?

Yes, "info threads" shows me only one thread.

[-- Attachment #2: Type: text/html, Size: 752 bytes --]