bug#65419: [Shepherd] Non-reponding service control fiber

bug#65419: [Shepherd] Non-reponding service control fiber

[Ludovic Courtès]

Hello,

On milano-guix-1 (a build machine behind bayfront, running shepherd
0.10.2), ‘herd status’ and ‘herd status guix-build-coordinator-agent’
would hang (there’s no ‘guix-build-coordinator’ process running).

‘herd stop childhurd2’ hangs and has no effect.

Conversely, ‘herd status nscd’ and similar for most other services works
fine.  When terminating a service’s process, the service gets respawned
just fine.

The conclusion seems to be that the control fiber of the ‘root’ service
is not responding: it is blocked on a get/put? did it exit?

Unfortunately we don’t have data from the logs that would give clues as
to what went wrong.

Ludo’.

bug#65419: [Shepherd] Non-reponding service control fiber

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1216 bytes --]

Hello,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

[...]

> The conclusion seems to be that the control fiber of the ‘root’ service
> is not responding: it is blocked on a get/put? did it exit?
>
> Unfortunately we don’t have data from the logs that would give clues as
> to what went wrong.

I've had a look at /var/log/messages but nothing seems wrong except
messages like this one:

--8<---------------cut here---------------start------------->8---

Aug 21 14:48:42 localhost shepherd[1]: 6 connections still in use after sshd-13752 termination. 
Aug 21 14:48:42 localhost shepherd[1]: Service sshd-13752 (PID 29977) exited with 255. 
Aug 21 14:48:42 localhost shepherd[1]: Service sshd-13752 has been disabled. 
Aug 21 14:48:42 localhost shepherd[1]: Transient service sshd-13752 terminated, now unregistered. 

--8<---------------cut here---------------end--------------->8---

Is it useful configuring the monitoring service [1] on milano-guix-1 to
have useful data in the logs in case we get a similar issue?

Thanks, Gio'


[1] https://www.gnu.org/software/shepherd/manual/shepherd.html#Monitoring-Service

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

bug#65419: [Shepherd] Non-reponding service control fiber

[Ludovic Courtès]

Hi,

Giovanni Biscuolo <g@xelera.eu> skribis:

> I've had a look at /var/log/messages but nothing seems wrong except
> messages like this one:
>
>
> Aug 21 14:48:42 localhost shepherd[1]: 6 connections still in use after sshd-13752 termination. 
> Aug 21 14:48:42 localhost shepherd[1]: Service sshd-13752 (PID 29977) exited with 255. 
> Aug 21 14:48:42 localhost shepherd[1]: Service sshd-13752 has been disabled. 
> Aug 21 14:48:42 localhost shepherd[1]: Transient service sshd-13752 terminated, now unregistered. 

Yeah, I think it happened earlier but unfortunately the previously logs
got deleted (rottlog is not behaving as expected).

> Is it useful configuring the monitoring service [1] on milano-guix-1 to
> have useful data in the logs in case we get a similar issue?

It wouldn’t help in this case, but it’s still interesting to have it
around.

  sudo herd eval root '(begin (use-modules (shepherd service monitoring)) (register-services (list (monitoring-service))))'
  sudo herd start monitoring

Ludo’.