bug#26976: On Hydra, offload crashes while trying to build linux-libre source

[ludo@gnu.org (Ludovic Courtès)]

Hi,

ludo@gnu.org (Ludovic Courtès) skribis:

> I was able to reproduce it on hydra.gnu.org with:
>
>   guix copy --to=hydra.gnunet.org /gnu/store/gi7r1v65zqhh8riqprq8nchfc9v9k156-guix-current
>
> which leads most of the time to SIGSEGV (I couldn’t get the SIGSEGV on
> my laptop):

Data from a recent run (libssh 0.7.5):

--8<---------------cut here---------------start------------->8---
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7f53f1b27700 (LWP 20388) 0x00007f53f013bac7 in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
  2    Thread 0x7f53f59bcf00 (LWP 20384) 0x00007f53f013b97e in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
  3    Thread 0x7f53f28cb700 (LWP 20387) 0x00007f53f4ff458f in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
  4    Thread 0x7f53f38cd700 (LWP 20385) 0x00007f53f4ff458f in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
  5    Thread 0x7f53ef2fb700 (LWP 20391) 0x00007f53f4ff7aed in read () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
  6    Thread 0x7f53f30cc700 (LWP 20386) 0x00007f53f4ff458f in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
(gdb) bt
#0  0x00007f53f013bac7 in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#1  0x00007f53f013d78d in deflate () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#2  0x00007f53ee459531 in gzip_compress (session=session@entry=0x1f3a7d0, source=source@entry=0x1f2b370, level=<optimized out>) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:85
#3  0x00007f53ee459863 in compress_buffer (session=session@entry=0x1f3a7d0, buf=0x1f2b370) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:106
#4  0x00007f53ee43ef87 in packet_send2 (session=session@entry=0x1f3a7d0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:535
#5  0x00007f53ee43f8d5 in packet_send (session=session@entry=0x1f3a7d0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:604
#6  0x00007f53ee430f4a in ssh_channel_send_eof (channel=0x1f44960) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1085
#7  0x00007f53ee431085 in ssh_channel_close (channel=0x1f44960) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1128
#8  0x00007f53ee89959f in ptob_close (channel=0x2451020) at channel-type.c:228
#9  0x00007f53f54fb785 in release_port (port=0x2451020) at ports.c:158
#10 0x00007f53f54fea0b in scm_close_port (port=0x2451020) at ports.c:887
#11 0x00007f53f553ac4d in vm_regular_engine (thread=0xfff51068, vp=0x1cacf30, registers=0x2080010, resume=-716695) at vm-engine.c:784
#12 0x00007f53f553e2aa in scm_call_n (proc=proc@entry=0x2408e40, argv=argv@entry=0x0, nargs=nargs@entry=0) at vm.c:1257
#13 0x00007f53f54c17d9 in scm_call_0 (proc=proc@entry=0x2408e40) at eval.c:481
#14 0x00007f53f552d4e6 in catch (tag=tag@entry=0x404, thunk=0x2408e40, handler=0x2408e20, pre_unwind_handler=0x4) at throw.c:137
#15 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>, pre_unwind_handler=<optimized out>) at throw.c:254
#16 0x00007f53f552d9df in scm_c_catch (tag=tag@entry=0x404, body=body@entry=0x7f53f54fea60 <do_close>, body_data=<optimized out>, handler=<optimized out>, handler_data=handler_data@entry=0x0, pre_unwind_handler=pre_unwind_handler@entry=0x0, pre_unwind_handler_data=0x0) at throw.c:377
#17 0x00007f53f552d9ee in scm_internal_catch (tag=tag@entry=0x404, body=body@entry=0x7f53f54fea60 <do_close>, body_data=<optimized out>, handler=<optimized out>, handler_data=handler_data@entry=0x0) at throw.c:386
#18 0x00007f53f54fa0a4 in finalize_port (ptr=<optimized out>, data=<optimized out>) at ports.c:701
#19 0x00007f53f521a70f in GC_invoke_finalizers () from /gnu/store/j4p7cnxb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1
#20 0x00007f53f54cb099 in scm_run_finalizers () at finalizers.c:387
#21 0x00007f53f54cb0f5 in finalization_thread_proc (unused=<optimized out>) at finalizers.c:223
#22 0x00007f53f54bbcba in c_body (d=0x7f53f1b26e60) at continuations.c:422
#23 0x00007f53f553ac4d in vm_regular_engine (thread=0xfff51068, vp=0x1cacf30, registers=0x2080010, resume=-716695) at vm-engine.c:784
#24 0x00007f53f553e2aa in scm_call_n (proc=proc@entry=0x1c04460, argv=argv@entry=0x0, nargs=nargs@entry=0) at vm.c:1257
#25 0x00007f53f54c17d9 in scm_call_0 (proc=proc@entry=0x1c04460) at eval.c:481
#26 0x00007f53f552d4e6 in catch (tag=tag@entry=0x404, thunk=0x1c04460, handler=0x1c043e0, pre_unwind_handler=0x1c04360) at throw.c:137
#27 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>, pre_unwind_handler=<optimized out>) at throw.c:254
#28 0x00007f53f552d9df in scm_c_catch (tag=tag@entry=0x404, body=body@entry=0x7f53f54bbcb0 <c_body>, body_data=body_data@entry=0x7f53f1b26e60, handler=handler@entry=0x7f53f54bbf40 <c_handler>, handler_data=handler_data@entry=0x7f53f1b26e60, pre_unwind_handler=pre_unwind_handler@entry=0x7f53f54bbda0 <pre_unwind_handler>, pre_unwind_handler_data=0x1bd6c80) at throw.c:377
#29 0x00007f53f54bc2a0 in scm_i_with_continuation_barrier (body=body@entry=0x7f53f54bbcb0 <c_body>, body_data=body_data@entry=0x7f53f1b26e60, handler=handler@entry=0x7f53f54bbf40 <c_handler>, handler_data=handler_data@entry=0x7f53f1b26e60, pre_unwind_handler=pre_unwind_handler@entry=0x7f53f54bbda0 <pre_unwind_handler>, pre_unwind_handler_data=0x1bd6c80) at continuations.c:360
#30 0x00007f53f54bc335 in scm_c_with_continuation_barrier (func=<optimized out>, data=<optimized out>) at continuations.c:456
#31 0x00007f53f552c10c in with_guile (base=0x7f53f1b26ec0, data=0x7f53f1b26ef0) at threads.c:660
#32 0x00007f53f5221732 in GC_call_with_stack_base () from /gnu/store/j4p7cnxb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1
#33 0x00007f53f552c498 in scm_i_with_guile (dynamic_state=<optimized out>, data=<optimized out>, func=<optimized out>) at threads.c:703
#34 scm_with_guile (func=<optimized out>, data=<optimized out>) at threads.c:709
#35 0x00007f53f4fee454 in start_thread () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#36 0x00007f53f39b67bf in clone () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
(gdb) thread 2 
[Switching to thread 2 (Thread 0x7f53f59bcf00 (LWP 20384))]
#0  0x00007f53f013b97e in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
(gdb) bt
#0  0x00007f53f013b97e in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#1  0x00007f53f013d78d in deflate () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#2  0x00007f53ee459531 in gzip_compress (session=session@entry=0x1f3a7d0, source=source@entry=0x1f2b370, level=<optimized out>) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:85
#3  0x00007f53ee459863 in compress_buffer (session=session@entry=0x1f3a7d0, buf=0x1f2b370) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:106
#4  0x00007f53ee43ef87 in packet_send2 (session=session@entry=0x1f3a7d0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:535
#5  0x00007f53ee43f8d5 in packet_send (session=session@entry=0x1f3a7d0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:604
#6  0x00007f53ee42f628 in channel_write_common (channel=0x1f3cc20, data=data@entry=0x25af020, len=len@entry=65536, is_stderr=is_stderr@entry=0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1321
#7  0x00007f53ee431127 in ssh_channel_write (channel=<optimized out>, data=data@entry=0x25af020, len=len@entry=65536) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1370
#8  0x00007f53ee8995ee in write_to_channel_port (channel=0x24ce5a0, src=<optimized out>, start=<optimized out>, count=65536) at channel-type.c:180
#9  0x00007f53f54fe72c in scm_i_write_bytes (port=0x24ce5a0, src=0x25af000, start=0, count=65536) at ports.c:2842
#10 0x00007f53f5505de4 in scm_put_bytevector (port=0x24ce5a0, bv=0x25af000, start=0x2, count=<optimized out>) at r6rs-ports.c:614
#11 0x00007f53f553ac4d in vm_regular_engine (thread=0x15c0, vp=0x1bdaf30, registers=0xc665f, resume=5) at vm-engine.c:784
#12 0x00007f53f553e2aa in scm_call_n (proc=0x7f53f59b4030, argv=argv@entry=0x7fffa1436c28, nargs=nargs@entry=1) at vm.c:1257
#13 0x00007f53f54c28d7 in scm_primitive_eval (exp=0x1cd7dd0) at eval.c:662
#14 0x00007f53f54de6eb in scm_primitive_load (filename=<optimized out>) at load.c:123
#15 0x00007f53f553ac4d in vm_regular_engine (thread=0x15c0, vp=0x1bdaf30, registers=0xc665f, resume=5) at vm-engine.c:784
#16 0x00007f53f553e2aa in scm_call_n (proc=0x7f53f59b4030, argv=argv@entry=0x7fffa1436e78, nargs=nargs@entry=1) at vm.c:1257
#17 0x00007f53f54c28d7 in scm_primitive_eval (exp=exp@entry=0x1cbc4e0) at eval.c:662
#18 0x00007f53f54c2933 in scm_eval (exp=0x1cbc4e0, module_or_state=module_or_state@entry=0x1c9f140) at eval.c:696
#19 0x00007f53f550d9d0 in scm_shell (argc=6, argv=0x7fffa14374d8) at script.c:454
#20 0x00007f53f54d94ed in invoke_main_func (body_data=0x7fffa1437380) at init.c:340
#21 0x00007f53f54bbcba in c_body (d=0x7fffa14372c0) at continuations.c:422
#22 0x00007f53f553ac4d in vm_regular_engine (thread=0x15c0, vp=0x1bdaf30, registers=0xc665f, resume=5) at vm-engine.c:784
#23 0x00007f53f553e2aa in scm_call_n (proc=proc@entry=0x1ba97a0, argv=argv@entry=0x0, nargs=nargs@entry=0) at vm.c:1257
#24 0x00007f53f54c17d9 in scm_call_0 (proc=proc@entry=0x1ba97a0) at eval.c:481
#25 0x00007f53f552d4e6 in catch (tag=tag@entry=0x404, thunk=0x1ba97a0, handler=0x1ba9780, pre_unwind_handler=0x1ba9720) at throw.c:137
#26 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>, pre_unwind_handler=<optimized out>) at throw.c:254
#27 0x00007f53f552d9df in scm_c_catch (tag=tag@entry=0x404, body=body@entry=0x7f53f54bbcb0 <c_body>, body_data=body_data@entry=0x7fffa14372c0, handler=handler@entry=0x7f53f54bbf40 <c_handler>, handler_data=handler_data@entry=0x7fffa14372c0, pre_unwind_handler=pre_unwind_handler@entry=0x7f53f54bbda0 <pre_unwind_handler>, pre_unwind_handler_data=0x1bd6c80) at throw.c:377
#28 0x00007f53f54bc2a0 in scm_i_with_continuation_barrier (body=body@entry=0x7f53f54bbcb0 <c_body>, body_data=body_data@entry=0x7fffa14372c0, handler=handler@entry=0x7f53f54bbf40 <c_handler>, handler_data=handler_data@entry=0x7fffa14372c0, pre_unwind_handler=pre_unwind_handler@entry=0x7f53f54bbda0 <pre_unwind_handler>, pre_unwind_handler_data=0x1bd6c80) at continuations.c:360
#29 0x00007f53f54bc335 in scm_c_with_continuation_barrier (func=<optimized out>, data=<optimized out>) at continuations.c:456
#30 0x00007f53f552c10c in with_guile (base=0x7fffa1437320, data=0x7fffa1437350) at threads.c:660
#31 0x00007f53f5221732 in GC_call_with_stack_base () from /gnu/store/j4p7cnxb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1
#32 0x00007f53f552c498 in scm_i_with_guile (dynamic_state=<optimized out>, data=data@entry=0x7fffa1437350, func=func@entry=0x7f53f54d94d0 <invoke_main_func>) at threads.c:703
#33 scm_with_guile (func=func@entry=0x7f53f54d94d0 <invoke_main_func>, data=data@entry=0x7fffa1437380) at threads.c:709
#34 0x00007f53f54d9682 in scm_boot_guile (argc=argc@entry=6, argv=argv@entry=0x7fffa14374d8, main_func=main_func@entry=0x400ca0 <inner_main>, closure=closure@entry=0x0) at init.c:323
#35 0x0000000000400b40 in main (argc=6, argv=0x7fffa14374d8) at guile.c:101
--8<---------------cut here---------------end--------------->8---

So we have the finalization thread closing a channel of session
0x1f3a7d0 (which causes a write on the channel), and the main thread
writing to a channel of that same session.

If we look at src/gzip.c:95 in libssh, it starts like this:

--8<---------------cut here---------------start------------->8---
static ssh_buffer gzip_compress(ssh_session session,ssh_buffer source,int level){
  z_stream *zout = session->current_crypto->compress_out_ctx;
  void *in_ptr = buffer_get_rest(source);
--8<---------------cut here---------------end--------------->8---

AIUI, that means there’s one output compression buffer per session, and
it’s not thread-safe (in Guile 2.2 finalizers are called from a separate
thread.)

I think the fix, in Guile-SSH, is to associate each libssh object
(session, channel, etc.) with a mutex, and to protect all uses of the
libssh object by that mutex.

(Alternately we could change libssh to move its compression buffers to
channels instead of sessions, but that may not be enough.)

WDYT, Artyom?  Would you be able to work on this?  :-)

Thanks,
Ludo’.