bug#67538: Shepherd stops responding during "guix system reconfigure"

bug#67538: Shepherd stops responding during "guix system reconfigure"

[Timo Wilken]

I run Guix System on a remote server, and I've just had Shepherd hang itself
completely during a "guix system reconfigure" -- see the terminal log below.

This is the system Shepherd, i.e. PID 1, so it hanging is obviously not good.

I was debugging some nginx/certbot-related issues, which is the reason for the
many invocations of "guix system reconfigure/roll-back/switch-generation".

I have since force-restarted the machine through /proc/sysrq-trigger. If some
information is missing, it may have to wait until this happens to me again,
sorry!

--8<---------------cut here---------------start------------->8---
$ cd src/tw-channel
$ sudo guix system reconfigure -L . tw/system/X.scm
$ sudo guix system roll-back   # 76 -> 75
$ sudo guix system roll-back   # 75 -> 74
$ sudo guix system roll-back   # 74 -> 73
$ sudo guix system switch-generation 76
$ edit tw/system/X.scm
$ sudo guix system reconfigure -L . tw/system/X.scm --allow-downgrades
guix system: warning: moving channel 'tw' from 6f4319548e425318c057fce48a3b39ceee4dd2ee to unrelated commit 8449867c353192d0c8313d67b3a02549f941ec56
substitute: updating substitutes from 'https://[...]'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/ygrlqslg9f5jwv50vjya2wbkzcxi260n-system.drv
  /gnu/store/0lvgfm2k4bgs5m338fq5dh7j5n7bhbm5-activate.scm.drv
  /gnu/store/gshibgm70parbj4x6y1x42qvq7n8x7c9-activate-service.scm.drv
  /gnu/store/z228g6557hg6nmmg1x9i2yp60q6c43qx-nginx.conf.drv
  /gnu/store/5s28r9s2mawd423q2azi4cdf203fi4f4-provenance.drv
  /gnu/store/jziijs3yg8q78fyn853pdzybpp73d5rd-boot.drv
  /gnu/store/58da936wr8zf9501x3m39pqbi7n866b2-shepherd.conf.drv
  /gnu/store/fp8cxrqhp58wwsbi75k3369a1yx8ldxy-shepherd-nginx.go.drv
  /gnu/store/fyday4l5if26p6ghhra331iyj48s7f0p-shepherd-nginx.scm.drv
  /gnu/store/ri2j9rv5d19x98ig1mc7yc3mpiknv88n-grub.cfg.drv

building /gnu/store/5s28r9s2mawd423q2azi4cdf203fi4f4-provenance.drv...
building /gnu/store/z228g6557hg6nmmg1x9i2yp60q6c43qx-nginx.conf.drv...
building /gnu/store/gshibgm70parbj4x6y1x42qvq7n8x7c9-activate-service.scm.drv...
building /gnu/store/fyday4l5if26p6ghhra331iyj48s7f0p-shepherd-nginx.scm.drv...
building /gnu/store/0lvgfm2k4bgs5m338fq5dh7j5n7bhbm5-activate.scm.drv...
building /gnu/store/fp8cxrqhp58wwsbi75k3369a1yx8ldxy-shepherd-nginx.go.drv...
building /gnu/store/58da936wr8zf9501x3m39pqbi7n866b2-shepherd.conf.drv...
building /gnu/store/jziijs3yg8q78fyn853pdzybpp73d5rd-boot.drv...
building /gnu/store/ygrlqslg9f5jwv50vjya2wbkzcxi260n-system.drv...
building /gnu/store/ri2j9rv5d19x98ig1mc7yc3mpiknv88n-grub.cfg.drv...
/gnu/store/6r0j6h4938hz5mddp61b61fw632dndzz-system
/gnu/store/253irqhvid0hkafig7ws4i81zmdsls37-grub.cfg

activating system...
The following derivation will be built:
  /gnu/store/wdpjdsxxkb2cyp2y9ffqwhkpf7ajb55k-switch-to-system.scm.drv

building /gnu/store/wdpjdsxxkb2cyp2y9ffqwhkpf7ajb55k-switch-to-system.scm.drv...
making '/gnu/store/6r0j6h4938hz5mddp61b61fw632dndzz-system' the current system...
setting up setuid programs in '/run/setuid-programs'...
populating /etc from /gnu/store/9yypp4bzsfprdq4vwjcj3f9jcj5dldk3-etc...
`/gnu/store/vxwqfm0fb8nj9flz272iwx8nwa82dsx4-openssh-authorized-keys/[x]' -> `/etc/ssh/authorized_keys.d/[x]'
`/gnu/store/vxwqfm0fb8nj9flz272iwx8nwa82dsx4-openssh-authorized-keys/[y]' -> `/etc/ssh/authorized_keys.d/[y]'
/var/lib/certbot/renew-certificates may need to be run
creating nginx log directory '/var/log/nginx'
creating nginx run directory '/var/run/nginx'
creating nginx temp directories '/var/run/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}_temp'
nginx: the configuration file /gnu/store/hkldki7rxg82i9nb3flsq6x58h81p2qr-nginx.conf syntax is ok
nginx: configuration file /gnu/store/hkldki7rxg82i9nb3flsq6x58h81p2qr-nginx.conf test is successful
The following derivation will be built:
  /gnu/store/dnz992gzxhpaq7xjcakdi53rdannsimf-install-bootloader.scm.drv

building /gnu/store/dnz992gzxhpaq7xjcakdi53rdannsimf-install-bootloader.scm.drv...
guix system: bootloader successfully installed on '(/boot/efi)'
The following derivation will be built:
  /gnu/store/q76v0yrh6vnbjiq2fw236lvn5mc2nl32-upgrade-shepherd-services.scm.drv

building /gnu/store/q76v0yrh6vnbjiq2fw236lvn5mc2nl32-upgrade-shepherd-services.scm.drv...
shepherd: Removing service 'fcgiwrap'...
shepherd: Done.
[ at this point, the process hangs ]
--8<---------------cut here---------------end--------------->8---

The mentioned derivation
/gnu/store/q76v0yrh6vnbjiq2fw236lvn5mc2nl32-upgrade-shepherd-services.scm.drv
builds the following upgrade-shepherd-services.scm:

--8<---------------cut here---------------start------------->8---
;; /gnu/store/qi2g4figwfn44nrlsaxgjn4p9sps6qv8-upgrade-shepherd-services.scm
[ %load-path mangling omitted ]
(begin
  (use-modules (gnu services herd) (srfi srfi-1))
  (parameterize
    ((shepherd-message-port (%make-void-port "w")))
    (load-services/safe
      '("/gnu/store/855sxj4rkzq41ypp39q5z3qgpbzgy6i7-shepherd-file-systems.scm"
        "/gnu/store/3vmb6s4wcxn1kw4388knzi6cpi48yz7c-shepherd-user-file-systems.scm"
        "/gnu/store/ji7i741ckxdmvpj4rkldhv5xxdjxpbcn-shepherd-file-system--boot-efi.scm"
        "/gnu/store/9pg4xp2ja9kjlrnm7j4a62hr07hsm3wg-shepherd-file-system--var-backups.scm"
        "/gnu/store/13b2181gh9qwvmar5zwlh8vvzamd7agc-shepherd-file-system--var-data.scm"
        "/gnu/store/6bqzwfdbiqj0k4ry25732b5iggna84x1-shepherd-file-system--dev-pts.scm"
        "/gnu/store/i8lf3s32nihq66k87ghyk4xzbrdqrc3d-shepherd-file-system--sys-kernel-debug.scm"
        "/gnu/store/rbx4h30awmx1mmdhf3i5sl7bihxxf199-shepherd-file-system--dev-shm.scm"
        "/gnu/store/r73iq9cm3jca3lppfr4dbzxrdppmbyxi-shepherd-file-system--sys-firmware-efi-efivars.scm"
        "/gnu/store/hy3c2bzpqyw4fjmkkpwm129dl190q1bl-shepherd-file-system--gnu-store.scm"
        "/gnu/store/spdzdw9zd10mjg0lw1ifbjrf7f1gzd4g-shepherd-root-file-system.scm"
        "/gnu/store/9xd2x4p0yf2ldxmalh6yv83dfxhx6rz7-shepherd-user-processes.scm"
        "/gnu/store/jhah6x7spssyi5nzydf1hs38nqa0vfp0-shepherd-host-name.scm"
        "/gnu/store/xjgk0d8ngxlkajv2w04ana4b1ib4jvk0-shepherd-user-homes.scm"
        "/gnu/store/xzn8j3gqs7rq6mp47gbagjslivwd02mg-shepherd-pam.scm"
        "/gnu/store/9lk622pwjbasiwi7zddk934pidp13cdc-shepherd-sysctl.scm"
        "/gnu/store/g0n4mlv609mgm4qy78rk134bw1bhly63-shepherd-udev.scm"
        "/gnu/store/wam791y3vj8qm76gi9ig6xyj0hhcj1mm-shepherd-nscd.scm"
        "/gnu/store/3wqlshnwf351mnk9xk6yxwf0rrlkvgk6-shepherd-guix-daemon.scm"
        "/gnu/store/5xa2i1km0ssnk3dcmxfqwanxc932mzki-shepherd-urandom-seed.scm"
        "/gnu/store/c4hf75wf5fl2jydcqnva86p6gmj0k42x-shepherd-loopback.scm"
        "/gnu/store/vbn734scz28hr2w7s5c3d5s9iwrjvr7f-shepherd-term-tty6.scm"
        "/gnu/store/fykgc2858d85v88yg8nscnax1sca7isd-shepherd-term-tty5.scm"
        "/gnu/store/42ahwwdz9xr3idimfxqhq2dz8z503sx8-shepherd-term-tty4.scm"
        "/gnu/store/7ar0dyx3a9wywx7jq98krcr7qwqla24w-shepherd-term-tty3.scm"
        "/gnu/store/8lavcb8j4ka56gr34yfd70r87c500dkw-shepherd-term-tty2.scm"
        "/gnu/store/zw340ia655wj0d6zp13367i8x19qshxl-shepherd-term-tty1.scm"
        "/gnu/store/qs4gb2mfvj5pibr7c3dx0s0nr0lr19v4-shepherd-term-console.scm"
        "/gnu/store/vwaw1dh8a8586f881j0sf53d5vybzwyx-shepherd-syslogd.scm"
        "/gnu/store/hfaqh39gg6r0r2sd1b3r9ayahhy2ziqv-shepherd-console-font-tty1.scm"
        "/gnu/store/ch4bw5h8ks3ksbp1949aqpn1vbsqvy2k-shepherd-console-font-tty2.scm"
        "/gnu/store/yqgcx129chxcx8wv1vwpvvd05iqgqlyw-shepherd-console-font-tty3.scm"
        "/gnu/store/pxbywc0ivqfd5gncrby5pw52lda5pqay-shepherd-console-font-tty4.scm"
        "/gnu/store/rfxbsv7cfp9zcskimljh52h5bz1ask62-shepherd-console-font-tty5.scm"
        "/gnu/store/q443gll57ca462a7zzl5yqaiw7l5g5yg-shepherd-console-font-tty6.scm"
        "/gnu/store/dpipig0pn7mh72ml4b2avbnij3cwz3p2-shepherd-virtual-terminal.scm"
        "/gnu/store/b4ndxky2waskw7yr5svq400dbwg6xdxs-shepherd-wireguard-wg0.scm"
        "/gnu/store/984irchd8kigp0cwv31jffwgvlcxcb6m-shepherd-ntpd.scm"
        "/gnu/store/rralzs460d1b5djwl80cc2s6nz6jd1nm-shepherd-networking.scm"
        "/gnu/store/515z3jc702qcsjqqrysm7j1nx6wrjrh3-shepherd-prometheus-node-exporter.scm"
        "/gnu/store/4600gcbra51adqkbc4rprvhnfwnz8ls0-shepherd-ssh-daemon-ssh-sshd.scm"
        "/gnu/store/pahrcqwxm7656r5g9njxs33ry6qncpmq-shepherd-php-fpm.scm"
        "/gnu/store/2wmvz4m7nbpri2dcksrsrmrj7i7fxg03-shepherd-mysql.scm"
        "/gnu/store/gf70rxc0b2myp84968c2dm3xhjmm5j66-shepherd-mysql-upgrade.scm"
        "/gnu/store/6v44ghv8wmm3q9a916bj6fdw6sqh8lmn-shepherd-syncthing-syncthing.scm"
        "/gnu/store/y5mmqzdb3j2imshbmn2bh95pbz10i08l-shepherd-transmission-daemon-transmission-bittorrent.scm"
        "/gnu/store/38lb7bwzm1g6nm3x9kgl628jrj8p2x1v-shepherd-git-daemon.scm"
        "/gnu/store/z8b6002pmlw72zxxd8cz7snh341h2nmq-shepherd-thermald.scm"
        "/gnu/store/d84dy6wa4z7ikwfjkd68s83idppfkgf6-shepherd-tor.scm"
        "/gnu/store/cwags534rgmig8lvynan15y1sp610g9q-shepherd-mcron.scm"
        "/gnu/store/pl3s5va584g1bira7k7swgzpb0qxq9g0-shepherd-nginx.scm")))
  (for-each unload-service '(fcgiwrap))
  (for-each start-service '(host-name user-homes sysctl mysql-upgrade nginx)))
--8<---------------cut here---------------end--------------->8---

Putting the above together with the output above, it seems that Shepherd
processes the (for-each unload-service '(fcgiwrap)) line fine, but gets stuck
before it can run (start-service 'host-name).

Here is the output of "guix describe". "tw" is my own personal channel, and
I've redacted one unofficial channel.

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 35	Nov 29 2023 00:49:37	(current)
  guix cd46757
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: cd46757c1a0f886848fbb6828c028dd2a2532767
  tw 8449867
    repository URL: [redacted]
    branch: master
    commit: 8449867c353192d0c8313d67b3a02549f941ec56
  [redacted channel] a7f7269
    repository URL: [redacted]
    branch: master
    commit: a7f7269f5f19c306c5082461b03797c2e92edf6f
--8<---------------cut here---------------end--------------->8---

Asking on IRC, I was pointed to https://issues.guix.gnu.org/66684. This issue
matches some of my symptoms (for me, Shepherd becomes completely unresponsive
while consuming 0% CPU), but I can't see any symptoms of the system time
having changed. I run the Guix-standard ntpd.

Here's the tail of /var/log/messages leading up to the hang. The first line
("shepherd[1]: Service nginx failed to start") is from the "sudo guix system
switch-generation 76" command. The following messages, up until "shepherd[1]:
Done." are from the final "sudo guix system reconfigure". The messages after
that are from other system services (Syncthing, mostly). All times are UTC+1.

--8<---------------cut here---------------start------------->8---
Nov 29 21:19:34 localhost shepherd[1]: Service nginx failed to start. 
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/hosts` was deleted, removing watch
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/hosts` was created, adding watch
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was deleted, removing watch
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was deleted, removing watch
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was created, adding watch
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was created, adding watch
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was written to
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/services` was deleted, removing watch
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/services` was created, adding watch
Nov 29 21:20:42 localhost nscd: 20914 monitored file `/etc/services` was written to
Nov 29 21:21:45 localhost nscd: 20914 monitored file `/etc/hosts` was deleted, removing watch
Nov 29 21:21:45 localhost nscd: 20914 monitored file `/etc/hosts` was created, adding watch
Nov 29 21:21:45 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was deleted, removing watch
Nov 29 21:21:45 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was deleted, removing watch
Nov 29 21:21:45 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was created, adding watch
Nov 29 21:21:45 localhost nscd: 20914 monitored file `/etc/nsswitch.conf` was created, adding watch
Nov 29 21:21:45 localhost nscd: 20914 monitored file `/etc/services` was deleted, removing watch
Nov 29 21:21:45 localhost nscd: 20914 monitored file `/etc/services` was created, adding watch
Nov 29 21:21:47 localhost shepherd[1]: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?). 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-systems. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for user-file-systems. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-system-/boot/efi. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-system-/var/backups.
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-system-/var/data. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-system-/dev/pts. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-system-/sys/kernel/debug. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-system-/dev/shm. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-system-/sys/firmware/efi/efivars. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for file-system-/gnu/store. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for root-file-system. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for user-processes. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for pam. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for udev. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for nscd. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for guix-daemon. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for urandom-seed. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for loopback. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for term-tty6. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for term-tty5. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for term-tty4. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for term-tty3. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for term-tty2. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for term-tty1. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for term-console. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for syslogd. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for console-font-tty1. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for console-font-tty2. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for console-font-tty3. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for console-font-tty4. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for console-font-tty5. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for console-font-tty6. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for virtual-terminal. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for wireguard-wg0. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for ntpd. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for networking. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for prometheus-node-exporter. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for ssh-daemon. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for php-fpm. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for mysql. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for syncthing-syncthing. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for transmission-daemon. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for git-daemon. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for thermald. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for tor. 
Nov 29 21:21:47 localhost shepherd[1]: Recording replacement for mcron. 
Nov 29 21:21:47 localhost shepherd[1]: Removing service 'fcgiwrap'... 
Nov 29 21:21:47 localhost shepherd[1]: Stopping service fcgiwrap... 
Nov 29 21:21:47 localhost shepherd[1]: Service fcgiwrap stopped. 
Nov 29 21:21:47 localhost shepherd[1]: Service fcgiwrap is now stopped. 
Nov 29 21:21:47 localhost shepherd[1]: Done. 
Nov 29 21:22:03 localhost shepherd[1]: Accepted connection on 0.0.0.0:[syncthing port] from [...]
Nov 29 21:23:23 localhost shepherd[1]: [syncthing] [P7WXJ] INFO: Lost primary connection to [...]
Nov 29 21:23:23 localhost shepherd[1]: [syncthing] [P7WXJ] INFO: Connection to [...]
Nov 29 21:23:24 localhost shepherd[1]: [syncthing] [P7WXJ] INFO: Established secure connection to [...]
Nov 29 21:23:24 localhost shepherd[1]: [syncthing] [P7WXJ] INFO: Device [...]
Nov 29 21:42:05 localhost -- MARK --
Nov 29 21:49:09 localhost shepherd[1]: [syncthing] [P7WXJ] INFO: Lost primary connection to [...]
Nov 29 21:49:09 localhost shepherd[1]: [syncthing] [P7WXJ] INFO: Connection to [...]
Nov 29 21:49:44 localhost shepherd[1]: [syncthing] [P7WXJ] INFO: Established secure connection to [...]
Nov 29 21:49:44 localhost shepherd[1]: [syncthing] [P7WXJ] INFO: Device [...]
Nov 29 22:02:05 localhost -- MARK --
--8<---------------cut here---------------end--------------->8---

bug#67538: Shepherd stops responding during "guix system reconfigure"

[Attila Lendvai]

i've also experienced this, and someone else on IRC also described the same behavior. that makes three of us.

i don't think it's relevant, but i'm also using syncthing.

my suspicion is that it's due to some error coming from a start GEXP that somehow derails shepherd's event loop.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
The use of power is only needed when you want to do something harmful, otherwise love is enough to get everything done.

bug#67538: Shepherd stops responding during "guix system reconfigure"

[Michal Atlas]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

I've been experiencing this occasionally ever since I started using guix. It happens on all of my machines and across different setups.

I don't use syncthing so it's probably not that.

The set of services I have has changed so much I'd have trouble identifying a singular service that was in all of the instances of this occurring save for those in %desktop-services.

iirc I once managed to get a debugger out when it happened and it's stuck waiting in one of the epoll/select/alike calls, sadly don't remember exactly which one. Will investigate next time it happens.

The only remedy I know is indeed SysRq.

So that's at least 4 people now

[-- Attachment #2: Type: text/html, Size: 738 bytes --]

bug#67538: Shepherd stops responding during "guix system reconfigure"

[Attila Lendvai]

> > my suspicion is that it's due to some error coming from a start
> > GEXP that somehow derails shepherd's event loop.


[...]


> iirc I once managed to get a debugger out when it happened and it's
> stuck waiting in one of the epoll/select/alike calls,


...or one of the start/stop GEXP's calls something that (sometimes?) blocks indefinitely (which violates the API of shepherd).

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
Child labor was not abolished, it was changed from productive work to counter-productive brainwashing, and made universal: compulsory public schooling.

bug#67538: Shepherd stops responding during "guix system reconfigure"

[Simon Streit]

Attila Lendvai <attila@lendvai.name> writes:

> i've also experienced this, and someone else on IRC also described the
> same behavior. that makes three of us.

I can confirm it too.  It looks as if shepherd hangs when something
changes.  But it doesn't happen all the time.  It also happens sometimes
when configuring a home environment.  There it is not much of a problem
to restart shepherd.

But that doesn't work on a system level.  There I have to hard reset the
system.

> i don't think it's relevant, but i'm also using syncthing.

I'm not sure about this either.  I've noticed too that the service
syncthing will sometimes hang when stopping or restarting.  This could
be a different issue with the service itself.

While thinking of it:  I don't use syncthing in my system declaration
anymore.  It move over to my home environment.  The hanging still
happens.


Regards

-- 
Simon

bug#67230: Shepherd stops responding during "guix system reconfigure"

[Timo Wilken]

After a bit of searching, it looks like 67538, 67230 and 65178 may be the same
issue.

Attila Lendvai wrote:
> > > my suspicion is that it's due to some error coming from a start
> > > GEXP that somehow derails shepherd's event loop.
> >
> > iirc I once managed to get a debugger out when it happened and it's
> > stuck waiting in one of the epoll/select/alike calls,
>
> ...or one of the start/stop GEXP's calls something that (sometimes?) blocks
> indefinitely (which violates the API of shepherd).

Same symptoms here again.

For context: this time I was trying to deploy some OCI/Docker containers using
Guix' `oci-container-service-type', specifically a Shepherd service called
"conduit". My code is here:

https://cgit.twilken.net/dotfiles/log/?h=matrix-containers

(Specifically, commits bf94f7872a1df293bd904bbd2c1ef7229f4f98a8 and
c87dcdae79c6266ac3dac70af08fbef5eb21629b.)

This is with Guix commit 1b2505217cf222d98cc960b8510660976a01cfa1.

I first ran "guix system reconfigure -L . tw/system/lud.scm" with commit
bf94f7872a1df293bd904bbd2c1ef7229f4f98a8, which had a bug (an env var was
wrong, so the container failed to start). This worked as expected in that
Shepherd tried to start the service, which failed, so Shepherd disabled it.

Then, I fixed the env var and re-ran "guix system reconfigure -L .
tw/system/lud.scm" with commit c87dcdae79c6266ac3dac70af08fbef5eb21629b.
Shepherd loaded the new "conduit" service fine, as far as I can tell, but
didn't restart it because it was still disabled.

I then enabled and started the service manually. Enabling worked fine, but on
start, I got no terminal output from Shepherd, and it hung.

I still had an error in my setup (directory permissions were wrong), and I got
a message in /var/log/messages to that effect:

--8<---------------cut here---------------start------------->8---
Dec 14 21:33:50 localhost shepherd[1]: Service conduit is currently disabled. 
Dec 14 21:34:04 localhost shepherd[1]: Enabled service conduit. 
Dec 14 21:34:07 localhost shepherd[1]: Starting service user-homes... 
Dec 14 21:34:07 localhost shepherd[1]: Service user-homes has been started. 
Dec 14 21:34:07 localhost shepherd[1]: Service user-homes started. 
Dec 14 21:34:07 localhost shepherd[1]: Service user-homes running with value #t. 
Dec 14 21:34:07 localhost shepherd[1]: Starting service conduit... 
Dec 14 21:34:07 localhost shepherd[1]: Service conduit has been started. 
Dec 14 21:34:07 localhost shepherd[1]: Service conduit started. 
Dec 14 21:34:07 localhost shepherd[1]: Service conduit running with value 13226. 
Dec 14 21:34:07 localhost shepherd[1]: [docker] conduit: [...] "IO error: While open a file for appending: /var/lib/matrix-conduit/LOG: Permission denied"
--8<---------------cut here---------------end--------------->8---

...showing that Shepherd had at least tried to start the new container. The
container is not running, though (due to the error shown above), and nothing
with PID 13226 is running.

The "herd start conduit" command did not return, and ^C-ing it did not help.
Afterwards, every "herd" command also hung without any output.

Here are the last four lines of the output of "sudo strace -s1000 herd status"
on such a hung machine:

--8<---------------cut here---------------start------------->8---
connect(10, {sa_family=AF_UNIX, sun_path="/var/run/shepherd/socket"}, 26) = 0
getcwd("/home/timo", 100)               = 11
write(10, "(shepherd-command (version 0) (action status) (service root) (arguments ()) (directory \"/home/timo\"))", 101) = 101
read(10,
--8<---------------cut here---------------end--------------->8---

The "read(10, " call never completes.

At least in this case, Shepherd still seems to be processing inbound inet
connections, so I can open new SSH connections to the machine.

Attaching to PID 1 with strace shows it is stuck in "epoll_wait(13, "
(unsurprisingly, fd 13 points to "anon_inode:[eventpoll]"). Here's a backtrace
of all threads in "gdb -p 1":

--8<---------------cut here---------------start------------->8---
(gdb) info threads
  Id   Target Id                                     Frame 
* 1    Thread 0x7f786544c380 (LWP 1) "shepherd"      0x00007f7865552626 in epoll_wait ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  2    Thread 0x7f7864e16640 (LWP 186) "GC-marker-0" 0x00007f78654cf16a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  3    Thread 0x7f7864615640 (LWP 187) "GC-marker-1" 0x00007f78654cf16a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  4    Thread 0x7f7863e14640 (LWP 188) "GC-marker-2" 0x00007f78654cf16a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  5    Thread 0x7f78634c6640 (LWP 190) "shepherd"    0x00007f786554300c in read ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
(gdb) thread apply all bt

Thread 5 (Thread 0x7f78634c6640 (LWP 190) "shepherd"):
#0  0x00007f786554300c in read () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f7865a48cc7 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#2  0x00007f78659427d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f786594438c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f786594e83c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007f7865a65455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007f7865a4d570 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#7  0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#8  0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#9  0x00007f78659e5b3e in scm_call_with_unblocked_asyncs () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#10 0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#11 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#12 0x00007f7865a6b0f3 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007f78659e7e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007f78659e95ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007f7865a90092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007f7865a6be1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007f78659ea396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007f7865a6b049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#21 0x00007f786594e7fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#22 0x00007f7865a64c5d in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#23 0x00007f78654d23aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#24 0x00007f7865552f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 4 (Thread 0x7f7863e14640 (LWP 188) "GC-marker-2"):
#0  0x00007f78654cf16a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f78654d17e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007f7865948740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f7865948897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f78654d23aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007f7865552f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 3 (Thread 0x7f7864615640 (LWP 187) "GC-marker-1"):
#0  0x00007f78654cf16a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f78654d17e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007f7865948740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f7865948897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f78654d23aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007f7865552f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 2 (Thread 0x7f7864e16640 (LWP 186) "GC-marker-0"):
#0  0x00007f78654cf16a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f78654d17e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007f7865948740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f7865948897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f78654d23aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007f7865552f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 1 (Thread 0x7f786544c380 (LWP 1) "shepherd"):
#0  0x00007f7865552626 in epoll_wait () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f7862bb9335 in ?? () from /gnu/store/h4nsywbhn8b4qyh40fhykk3q40qkr3wd-guile-fibers-1.3.1/lib/guile/3.0/extensions/fibers-epoll.so
#2  0x00007f78659427d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f786594438c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f786594e83c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007f7865a65455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007f7862bb96ce in ?? () from /gnu/store/h4nsywbhn8b4qyh40fhykk3q40qkr3wd-guile-fibers-1.3.1/lib/guile/3.0/extensions/fibers-epoll.so
#7  0x00007f78606246c2 in ?? ()
#8  0x00007f78620ba628 in ?? ()
#9  0x00007f7860627610 in ?? ()
#10 0x00007f786520ad80 in ?? ()
#11 0x00007f7865a14edc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#12 0x00007f7865a71215 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007f78659e9977 in scm_primitive_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007f7865a1dff9 in scm_primitive_load () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007f78659e9977 in scm_primitive_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007f78659ef846 in scm_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007f7865a4e3e6 in scm_shell () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#21 0x00007f7865a008cc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#22 0x00007f78659e7e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#23 0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#24 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#25 0x00007f78659e95ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#26 0x00007f7865a90092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#27 0x00007f7865a6be1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#28 0x00007f78659ea396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#29 0x00007f7865a6b049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#30 0x00007f786594e7fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#31 0x00007f7865a653f8 in scm_with_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#32 0x00007f7865a098e5 in scm_boot_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#33 0x00000000004010f7 in ?? ()
#34 0x00007f78654761f7 in __libc_start_call_main () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#35 0x00007f78654762ac in __libc_start_main_impl () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#36 0x0000000000401171 in ?? ()
--8<---------------cut here---------------end--------------->8---

Unrelatedly, I also have another Shepherd on a different machine that became
stuck after I ran a bunch of "guix system reconfigure" commands. The
backtraces there, if it helps:

--8<---------------cut here---------------start------------->8---
(gdb) info threads 
  Id   Target Id                                     Frame 
* 1    Thread 0x7ffaceef2380 (LWP 1) "shepherd"      0x00007fface938626 in epoll_wait ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  2    Thread 0x7fface1aa640 (LWP 231) "GC-marker-0" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  3    Thread 0x7ffacd9a9640 (LWP 232) "GC-marker-1" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  4    Thread 0x7ffacd1a8640 (LWP 233) "GC-marker-2" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  5    Thread 0x7ffacc9a7640 (LWP 234) "GC-marker-3" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  6    Thread 0x7ffacc1a6640 (LWP 235) "GC-marker-4" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  7    Thread 0x7ffacb9a5640 (LWP 236) "GC-marker-5" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  8    Thread 0x7ffacb1a4640 (LWP 237) "GC-marker-6" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  9    Thread 0x7ffaca832640 (LWP 249) "shepherd"    0x00007fface92900c in read ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  10   Thread 0x7ffac89ca640 (LWP 26693) "shepherd"  0x00007fface92900c in read ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
(gdb) thread apply all bt

Thread 10 (Thread 0x7ffac89ca640 (LWP 26693) "shepherd"):
#0  0x00007fface92900c in read () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007ffacedf0e57 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#2  0x00007ffaced3c7d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced3e38c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007ffaced4883c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007ffacee62455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007ffacedf903d in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#7  0x00007ffacede4e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#8  0x00007ffac6832022 in ?? ()
#9  0x00007fface4d97f0 in ?? ()
#10 0x00007ffac94766c0 in ?? ()
#11 0x00007fface5f4b40 in ?? ()
#12 0x00007ffacee11edc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007ffacee6e215 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007ffacede65ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007ffacee8d092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007ffacee68e1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007ffacede7396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007ffacee68049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007ffaced487fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#21 0x00007ffacee623f8 in scm_with_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#22 0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#23 0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 9 (Thread 0x7ffaca832640 (LWP 249) "shepherd"):
#0  0x00007fface92900c in read () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007ffacee45cc7 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#2  0x00007ffaced3c7d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced3e38c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007ffaced4883c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007ffacee62455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007ffacee4a570 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#7  0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#8  0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#9  0x00007ffacede2b3e in scm_call_with_unblocked_asyncs () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#10 0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#11 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#12 0x00007ffacee680f3 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007ffacede4e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007ffacede65ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007ffacee8d092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007ffacee68e1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007ffacede7396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007ffacee68049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#21 0x00007ffaced487fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#22 0x00007ffacee61c5d in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#23 0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#24 0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 8 (Thread 0x7ffacb1a4640 (LWP 237) "GC-marker-6"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 7 (Thread 0x7ffacb9a5640 (LWP 236) "GC-marker-5"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--c
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 6 (Thread 0x7ffacc1a6640 (LWP 235) "GC-marker-4"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 5 (Thread 0x7ffacc9a7640 (LWP 234) "GC-marker-3"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 4 (Thread 0x7ffacd1a8640 (LWP 233) "GC-marker-2"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 3 (Thread 0x7ffacd9a9640 (LWP 232) "GC-marker-1"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 2 (Thread 0x7fface1aa640 (LWP 231) "GC-marker-0"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 1 (Thread 0x7ffaceef2380 (LWP 1) "shepherd"):
#0  0x00007fface938626 in epoll_wait () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007ffac9efc335 in ?? () from /gnu/store/h4nsywbhn8b4qyh40fhykk3q40qkr3wd-guile-fibers-1.3.1/lib/guile/3.0/extensions/fibers-epoll.so
#2  0x00007ffaced3c7d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced3e38c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007ffaced4883c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007ffacee62455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007ffac9efc6ce in ?? () from /gnu/store/h4nsywbhn8b4qyh40fhykk3q40qkr3wd-guile-fibers-1.3.1/lib/guile/3.0/extensions/fibers-epoll.so
#7  0x00007ffac76416c2 in ?? ()
#8  0x00007ffac934f594 in ?? ()
#9  0x00007ffac9476d83 in ?? ()
#10 0x00007fface5f4d80 in ?? ()
#11 0x00007ffacee11edc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#12 0x00007ffacee6e652 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007ffacede6977 in scm_primitive_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007ffacee1aff9 in scm_primitive_load () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007ffacede6977 in scm_primitive_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007ffacedec846 in scm_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007ffacee4b3e6 in scm_shell () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#21 0x00007ffacedfd8cc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#22 0x00007ffacede4e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#23 0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#24 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#25 0x00007ffacede65ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#26 0x00007ffacee8d092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#27 0x00007ffacee68e1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#28 0x00007ffacede7396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#29 0x00007ffacee68049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#30 0x00007ffaced487fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#31 0x00007ffacee623f8 in scm_with_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#32 0x00007ffacee068e5 in scm_boot_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#33 0x00000000004010f7 in ?? ()
#34 0x00007fface85c1f7 in __libc_start_call_main () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#35 0x00007fface85c2ac in __libc_start_main_impl () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#36 0x0000000000401171 in ?? ()
--8<---------------cut here---------------end--------------->8---

bug#65178: Shepherd stops responding during "guix system reconfigure"

[Attila Lendvai]

i think i have found the root cause of this, as documented here: https://issues.guix.gnu.org/67839

that issue contains patches for shepherd to reproduce it in its test suite.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“What divides libertarians from everybody else is not a belief about rights or what rights people have, because the judgments libertarians make about the state are the same as the judgments almost everyone makes about private agents. So it's not that we believe in rights that other people don't believe in, or that other people believe in rights that we don't believe in. It's that other people think the state is exempt from the moral principles that apply to non-government agents.”
	— Michael Huemer

bug#67230: Shepherd stops responding during "guix system reconfigure"

[Timo Wilken]

On Fri Dec 15, 2023 at 8:47 PM CET, Attila Lendvai wrote:
> i think i have found the root cause of this, as documented here: https://issues.guix.gnu.org/67839
>
> that issue contains patches for shepherd to reproduce it in its test suite.

Thank you very much for this, Attila!

Are the patch in 67839 and/or your branch "attila" linked from there in a
state that I could test them locally? Would it be valuable to you if I ran a
patched Shepherd and sent logs and/or backtraces as I encountered them?

bug#67538: Shepherd stops responding during "guix system reconfigure"

[Attila Lendvai]

> Thank you very much for this, Attila!


you're welcome! :)


> Are the patch in 67839 and/or your branch "attila" linked from there in a
> state that I could test them locally? Would it be valuable to you if I ran a
> patched Shepherd and sent logs and/or backtraces as I encountered them?


it's nice of you, but not really. now that we have a failing test case in shepherd's unit tests that can reproduce it much easier.

with #67839 you would only get you an extra "Assertion failed" message over master, without much useful output.

as for my branch, it would emit a lot of useful log, including backtraces, but i keep force-pushing into it. i'm running my servers with it, though, so if you feel really adventurous, and want to join the debugging, then you can try... otherwise it's too much in flux.

what we need to focus on now is making shepherd's test suite run clean again, one way or another. then i can test it in a real life environment, and report back with any possible findings.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Ignorance might be bliss for the ignorant, but for the rest of us it's a fucking pain in the ass.”
	— Ricky Gervais

bug#65419: [Shepherd] Non-responding service control fiber

[Ludovic Courtès]

Hello,

Attila Lendvai <attila@lendvai.name> skribis:

> i think i have found the root cause of this, as documented here: https://issues.guix.gnu.org/67839
>
> that issue contains patches for shepherd to reproduce it in its test suite.

Yes, it looks like this long-standing and hard-to-debug issue may well
be fixed now, thumbs up Attila!!

We have accumulated quite a few fixes by now so I think I’ll release
0.10.3 hopefully in 2023 and otherwise soon after.

Thanks,
Ludo’.