From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#26976: On Hydra, offload crashes while trying to build linux-libre source Date: Mon, 05 Jun 2017 23:33:27 +0200 Message-ID: <87k24q86hk.fsf@gnu.org> References: <87h90japz0.fsf@netris.org> <87h90h966f.fsf@gnu.org> <87pof5dnja.fsf@gnu.org> <87mva88oe9.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:47095) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dHzdV-0005Qz-Oy for bug-guix@gnu.org; Mon, 05 Jun 2017 17:34:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dHzdS-00028G-Ig for bug-guix@gnu.org; Mon, 05 Jun 2017 17:34:05 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:55493) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dHzdS-000285-Ef for bug-guix@gnu.org; Mon, 05 Jun 2017 17:34:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87mva88oe9.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Sat, 20 May 2017 00:36:46 +0200") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Mark H Weaver Cc: Artyom Poptsov , 26976@debbugs.gnu.org Hi, ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > I was able to reproduce it on hydra.gnu.org with: > > guix copy --to=3Dhydra.gnunet.org /gnu/store/gi7r1v65zqhh8riqprq8nchfc9= v9k156-guix-current > > which leads most of the time to SIGSEGV (I couldn=E2=80=99t get the SIGSE= GV on > my laptop): Data from a recent run (libssh 0.7.5): --8<---------------cut here---------------start------------->8--- (gdb) info threads Id Target Id Frame=20 * 1 Thread 0x7f53f1b27700 (LWP 20388) 0x00007f53f013bac7 in deflate_fast= () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.s= o.1 2 Thread 0x7f53f59bcf00 (LWP 20384) 0x00007f53f013b97e in deflate_fast= () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.s= o.1 3 Thread 0x7f53f28cb700 (LWP 20387) 0x00007f53f4ff458f in pthread_cond= _wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glib= c-2.25/lib/libpthread.so.0 4 Thread 0x7f53f38cd700 (LWP 20385) 0x00007f53f4ff458f in pthread_cond= _wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glib= c-2.25/lib/libpthread.so.0 5 Thread 0x7f53ef2fb700 (LWP 20391) 0x00007f53f4ff7aed in read () from= /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0 6 Thread 0x7f53f30cc700 (LWP 20386) 0x00007f53f4ff458f in pthread_cond= _wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glib= c-2.25/lib/libpthread.so.0 (gdb) bt #0 0x00007f53f013bac7 in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qs= g607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1 #1 0x00007f53f013d78d in deflate () from /gnu/store/jwkcd7siv6fcyl0qsg607b= g9c8ap0gqr-zlib-1.2.11/lib/libz.so.1 #2 0x00007f53ee459531 in gzip_compress (session=3Dsession@entry=3D0x1f3a7d= 0, source=3Dsource@entry=3D0x1f2b370, level=3D) at /tmp/guix= -build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:85 #3 0x00007f53ee459863 in compress_buffer (session=3Dsession@entry=3D0x1f3a= 7d0, buf=3D0x1f2b370) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/sr= c/gzip.c:106 #4 0x00007f53ee43ef87 in packet_send2 (session=3Dsession@entry=3D0x1f3a7d0= ) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:535 #5 0x00007f53ee43f8d5 in packet_send (session=3Dsession@entry=3D0x1f3a7d0)= at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:604 #6 0x00007f53ee430f4a in ssh_channel_send_eof (channel=3D0x1f44960) at /tm= p/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1085 #7 0x00007f53ee431085 in ssh_channel_close (channel=3D0x1f44960) at /tmp/g= uix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1128 #8 0x00007f53ee89959f in ptob_close (channel=3D0x2451020) at channel-type.= c:228 #9 0x00007f53f54fb785 in release_port (port=3D0x2451020) at ports.c:158 #10 0x00007f53f54fea0b in scm_close_port (port=3D0x2451020) at ports.c:887 #11 0x00007f53f553ac4d in vm_regular_engine (thread=3D0xfff51068, vp=3D0x1c= acf30, registers=3D0x2080010, resume=3D-716695) at vm-engine.c:784 #12 0x00007f53f553e2aa in scm_call_n (proc=3Dproc@entry=3D0x2408e40, argv= =3Dargv@entry=3D0x0, nargs=3Dnargs@entry=3D0) at vm.c:1257 #13 0x00007f53f54c17d9 in scm_call_0 (proc=3Dproc@entry=3D0x2408e40) at eva= l.c:481 #14 0x00007f53f552d4e6 in catch (tag=3Dtag@entry=3D0x404, thunk=3D0x2408e40= , handler=3D0x2408e20, pre_unwind_handler=3D0x4) at throw.c:137 #15 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=3Dkey@entr= y=3D0x404, thunk=3D, handler=3D, pre_unwind_h= andler=3D) at throw.c:254 #16 0x00007f53f552d9df in scm_c_catch (tag=3Dtag@entry=3D0x404, body=3Dbody= @entry=3D0x7f53f54fea60 , body_data=3D, handler=3D= , handler_data=3Dhandler_data@entry=3D0x0, pre_unwind_handle= r=3Dpre_unwind_handler@entry=3D0x0, pre_unwind_handler_data=3D0x0) at throw= .c:377 #17 0x00007f53f552d9ee in scm_internal_catch (tag=3Dtag@entry=3D0x404, body= =3Dbody@entry=3D0x7f53f54fea60 , body_data=3D, han= dler=3D, handler_data=3Dhandler_data@entry=3D0x0) at throw.c= :386 #18 0x00007f53f54fa0a4 in finalize_port (ptr=3D, data=3D) at ports.c:701 #19 0x00007f53f521a70f in GC_invoke_finalizers () from /gnu/store/j4p7cnxb6= byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1 #20 0x00007f53f54cb099 in scm_run_finalizers () at finalizers.c:387 #21 0x00007f53f54cb0f5 in finalization_thread_proc (unused=3D) at finalizers.c:223 #22 0x00007f53f54bbcba in c_body (d=3D0x7f53f1b26e60) at continuations.c:422 #23 0x00007f53f553ac4d in vm_regular_engine (thread=3D0xfff51068, vp=3D0x1c= acf30, registers=3D0x2080010, resume=3D-716695) at vm-engine.c:784 #24 0x00007f53f553e2aa in scm_call_n (proc=3Dproc@entry=3D0x1c04460, argv= =3Dargv@entry=3D0x0, nargs=3Dnargs@entry=3D0) at vm.c:1257 #25 0x00007f53f54c17d9 in scm_call_0 (proc=3Dproc@entry=3D0x1c04460) at eva= l.c:481 #26 0x00007f53f552d4e6 in catch (tag=3Dtag@entry=3D0x404, thunk=3D0x1c04460= , handler=3D0x1c043e0, pre_unwind_handler=3D0x1c04360) at throw.c:137 #27 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=3Dkey@entr= y=3D0x404, thunk=3D, handler=3D, pre_unwind_h= andler=3D) at throw.c:254 #28 0x00007f53f552d9df in scm_c_catch (tag=3Dtag@entry=3D0x404, body=3Dbody= @entry=3D0x7f53f54bbcb0 , body_data=3Dbody_data@entry=3D0x7f53f1b26= e60, handler=3Dhandler@entry=3D0x7f53f54bbf40 , handler_data=3Dh= andler_data@entry=3D0x7f53f1b26e60, pre_unwind_handler=3Dpre_unwind_handler= @entry=3D0x7f53f54bbda0 , pre_unwind_handler_data=3D0x1= bd6c80) at throw.c:377 #29 0x00007f53f54bc2a0 in scm_i_with_continuation_barrier (body=3Dbody@entr= y=3D0x7f53f54bbcb0 , body_data=3Dbody_data@entry=3D0x7f53f1b26e60, = handler=3Dhandler@entry=3D0x7f53f54bbf40 , handler_data=3Dhandle= r_data@entry=3D0x7f53f1b26e60, pre_unwind_handler=3Dpre_unwind_handler@entr= y=3D0x7f53f54bbda0 , pre_unwind_handler_data=3D0x1bd6c8= 0) at continuations.c:360 #30 0x00007f53f54bc335 in scm_c_with_continuation_barrier (func=3D, data=3D) at continuations.c:456 #31 0x00007f53f552c10c in with_guile (base=3D0x7f53f1b26ec0, data=3D0x7f53f= 1b26ef0) at threads.c:660 #32 0x00007f53f5221732 in GC_call_with_stack_base () from /gnu/store/j4p7cn= xb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1 #33 0x00007f53f552c498 in scm_i_with_guile (dynamic_state=3D= , data=3D, func=3D) at threads.c:703 #34 scm_with_guile (func=3D, data=3D) at thre= ads.c:709 #35 0x00007f53f4fee454 in start_thread () from /gnu/store/rmjlycdgiq8pfy5hf= i42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0 #36 0x00007f53f39b67bf in clone () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3= k7p6kdav-glibc-2.25/lib/libc.so.6 (gdb) thread 2=20 [Switching to thread 2 (Thread 0x7f53f59bcf00 (LWP 20384))] #0 0x00007f53f013b97e in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qs= g607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1 (gdb) bt #0 0x00007f53f013b97e in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qs= g607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1 #1 0x00007f53f013d78d in deflate () from /gnu/store/jwkcd7siv6fcyl0qsg607b= g9c8ap0gqr-zlib-1.2.11/lib/libz.so.1 #2 0x00007f53ee459531 in gzip_compress (session=3Dsession@entry=3D0x1f3a7d= 0, source=3Dsource@entry=3D0x1f2b370, level=3D) at /tmp/guix= -build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:85 #3 0x00007f53ee459863 in compress_buffer (session=3Dsession@entry=3D0x1f3a= 7d0, buf=3D0x1f2b370) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/sr= c/gzip.c:106 #4 0x00007f53ee43ef87 in packet_send2 (session=3Dsession@entry=3D0x1f3a7d0= ) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:535 #5 0x00007f53ee43f8d5 in packet_send (session=3Dsession@entry=3D0x1f3a7d0)= at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:604 #6 0x00007f53ee42f628 in channel_write_common (channel=3D0x1f3cc20, data= =3Ddata@entry=3D0x25af020, len=3Dlen@entry=3D65536, is_stderr=3Dis_stderr@e= ntry=3D0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c= :1321 #7 0x00007f53ee431127 in ssh_channel_write (channel=3D, dat= a=3Ddata@entry=3D0x25af020, len=3Dlen@entry=3D65536) at /tmp/guix-build-lib= ssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1370 #8 0x00007f53ee8995ee in write_to_channel_port (channel=3D0x24ce5a0, src= =3D, start=3D, count=3D65536) at channel-type= .c:180 #9 0x00007f53f54fe72c in scm_i_write_bytes (port=3D0x24ce5a0, src=3D0x25af= 000, start=3D0, count=3D65536) at ports.c:2842 #10 0x00007f53f5505de4 in scm_put_bytevector (port=3D0x24ce5a0, bv=3D0x25af= 000, start=3D0x2, count=3D) at r6rs-ports.c:614 #11 0x00007f53f553ac4d in vm_regular_engine (thread=3D0x15c0, vp=3D0x1bdaf3= 0, registers=3D0xc665f, resume=3D5) at vm-engine.c:784 #12 0x00007f53f553e2aa in scm_call_n (proc=3D0x7f53f59b4030, argv=3Dargv@en= try=3D0x7fffa1436c28, nargs=3Dnargs@entry=3D1) at vm.c:1257 #13 0x00007f53f54c28d7 in scm_primitive_eval (exp=3D0x1cd7dd0) at eval.c:662 #14 0x00007f53f54de6eb in scm_primitive_load (filename=3D) a= t load.c:123 #15 0x00007f53f553ac4d in vm_regular_engine (thread=3D0x15c0, vp=3D0x1bdaf3= 0, registers=3D0xc665f, resume=3D5) at vm-engine.c:784 #16 0x00007f53f553e2aa in scm_call_n (proc=3D0x7f53f59b4030, argv=3Dargv@en= try=3D0x7fffa1436e78, nargs=3Dnargs@entry=3D1) at vm.c:1257 #17 0x00007f53f54c28d7 in scm_primitive_eval (exp=3Dexp@entry=3D0x1cbc4e0) = at eval.c:662 #18 0x00007f53f54c2933 in scm_eval (exp=3D0x1cbc4e0, module_or_state=3Dmodu= le_or_state@entry=3D0x1c9f140) at eval.c:696 #19 0x00007f53f550d9d0 in scm_shell (argc=3D6, argv=3D0x7fffa14374d8) at sc= ript.c:454 #20 0x00007f53f54d94ed in invoke_main_func (body_data=3D0x7fffa1437380) at = init.c:340 #21 0x00007f53f54bbcba in c_body (d=3D0x7fffa14372c0) at continuations.c:422 #22 0x00007f53f553ac4d in vm_regular_engine (thread=3D0x15c0, vp=3D0x1bdaf3= 0, registers=3D0xc665f, resume=3D5) at vm-engine.c:784 #23 0x00007f53f553e2aa in scm_call_n (proc=3Dproc@entry=3D0x1ba97a0, argv= =3Dargv@entry=3D0x0, nargs=3Dnargs@entry=3D0) at vm.c:1257 #24 0x00007f53f54c17d9 in scm_call_0 (proc=3Dproc@entry=3D0x1ba97a0) at eva= l.c:481 #25 0x00007f53f552d4e6 in catch (tag=3Dtag@entry=3D0x404, thunk=3D0x1ba97a0= , handler=3D0x1ba9780, pre_unwind_handler=3D0x1ba9720) at throw.c:137 #26 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=3Dkey@entr= y=3D0x404, thunk=3D, handler=3D, pre_unwind_h= andler=3D) at throw.c:254 #27 0x00007f53f552d9df in scm_c_catch (tag=3Dtag@entry=3D0x404, body=3Dbody= @entry=3D0x7f53f54bbcb0 , body_data=3Dbody_data@entry=3D0x7fffa1437= 2c0, handler=3Dhandler@entry=3D0x7f53f54bbf40 , handler_data=3Dh= andler_data@entry=3D0x7fffa14372c0, pre_unwind_handler=3Dpre_unwind_handler= @entry=3D0x7f53f54bbda0 , pre_unwind_handler_data=3D0x1= bd6c80) at throw.c:377 #28 0x00007f53f54bc2a0 in scm_i_with_continuation_barrier (body=3Dbody@entr= y=3D0x7f53f54bbcb0 , body_data=3Dbody_data@entry=3D0x7fffa14372c0, = handler=3Dhandler@entry=3D0x7f53f54bbf40 , handler_data=3Dhandle= r_data@entry=3D0x7fffa14372c0, pre_unwind_handler=3Dpre_unwind_handler@entr= y=3D0x7f53f54bbda0 , pre_unwind_handler_data=3D0x1bd6c8= 0) at continuations.c:360 #29 0x00007f53f54bc335 in scm_c_with_continuation_barrier (func=3D, data=3D) at continuations.c:456 #30 0x00007f53f552c10c in with_guile (base=3D0x7fffa1437320, data=3D0x7fffa= 1437350) at threads.c:660 #31 0x00007f53f5221732 in GC_call_with_stack_base () from /gnu/store/j4p7cn= xb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1 #32 0x00007f53f552c498 in scm_i_with_guile (dynamic_state=3D= , data=3Ddata@entry=3D0x7fffa1437350, func=3Dfunc@entry=3D0x7f53f54d94d0 ) at threads.c:703 #33 scm_with_guile (func=3Dfunc@entry=3D0x7f53f54d94d0 , = data=3Ddata@entry=3D0x7fffa1437380) at threads.c:709 #34 0x00007f53f54d9682 in scm_boot_guile (argc=3Dargc@entry=3D6, argv=3Darg= v@entry=3D0x7fffa14374d8, main_func=3Dmain_func@entry=3D0x400ca0 , closure=3Dclosure@entry=3D0x0) at init.c:323 #35 0x0000000000400b40 in main (argc=3D6, argv=3D0x7fffa14374d8) at guile.c= :101 --8<---------------cut here---------------end--------------->8--- So we have the finalization thread closing a channel of session 0x1f3a7d0 (which causes a write on the channel), and the main thread writing to a channel of that same session. If we look at src/gzip.c:95 in libssh, it starts like this: --8<---------------cut here---------------start------------->8--- static ssh_buffer gzip_compress(ssh_session session,ssh_buffer source,int l= evel){ z_stream *zout =3D session->current_crypto->compress_out_ctx; void *in_ptr =3D buffer_get_rest(source); --8<---------------cut here---------------end--------------->8--- AIUI, that means there=E2=80=99s one output compression buffer per session,= and it=E2=80=99s not thread-safe (in Guile 2.2 finalizers are called from a sep= arate thread.) I think the fix, in Guile-SSH, is to associate each libssh object (session, channel, etc.) with a mutex, and to protect all uses of the libssh object by that mutex. (Alternately we could change libssh to move its compression buffers to channels instead of sessions, but that may not be enough.) WDYT, Artyom? Would you be able to work on this? :-) Thanks, Ludo=E2=80=99.