bug#56519: Shepherd non-deterministically fails to load the "guix-daemon" service after the "user-processes" service has been started

[Markus Nilsson <markusnilsson890@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2976 bytes --]

Hi Ludo,

Thankyou for taking a look at this issue!

I think I didn't make it clear in the original bug report what the actual
problem was.

I understand that in the beginning of the boot process the
"device-mapping-streetkid_VG_storage-streetkid_LV_home" shepherd service is
failing and this is causing shepherd services that depend on it to fail to
start as a consequence. Later in the boot process the
"device-mapping-streetkid_VG_storage-streetkid_LV_home" does successfully
start though.

What happens is that after the
"device-mapping-streetkid_VG_storage-streetkid_LV_home" does start (after a
couple of tries), other shepherd services will then start including the
"user-processes" shepherd service. The problem is that the "guix-daemon"
shepherd service doesn't always start after that. Sometimes it will, but
sometimes it won't (see my first message for examples). Shouldn't the
"guix-daemon" shepherd service ALWAYS start after the "user-processes"
shepherd service starts?

I also found that the "nscd" shepherd service is failing to start even
though the "user-processes" shepherd service has started.

The following terminal output was taken after my streetkid server had
finished booting up. This should make things clearer:

1 mark@streetkid ~$ sudo herd status
2 Started:
3  + /proc/fs/nfsd
4  + console-font-tty1
5  + console-font-tty2
6  + console-font-tty3
7  + console-font-tty4
8  + console-font-tty5
9  + console-font-tty6
10  + device-mapping-streetkid_VG_storage-streetkid_LV_home
11  + file-system-/dev/pts
12  + file-system-/dev/shm
13  + file-system-/gnu/store
14  + file-system-/home
15  + file-system-/sys/firmware/efi/efivars
16  + file-system-/sys/kernel/debug
17  + file-systems
18  + idmap-daemon
19  + loopback
20  + mcron
21  + networking
22  + nfs
23  + ntpd
24  + root
25  + root-file-system
26  + rpc-pipefs
27  + rpc.mountd
28  + rpc.nfsd
29  + rpc.statd
30  + rpcbind-daemon
31  + ssh-daemon
32  + sshd-1
33  + syslogd
34  + term-tty1
35  + term-tty2
36  + term-tty3
37  + term-tty4
38  + term-tty5
39  + term-tty6
40  + udev
41  + urandom-seed
42  + user-file-systems
43  + user-processes
44  + virtual-terminal
45 Stopped:
46  - guix-daemon
47  - nscd
48  - term-console
49 One-shot:
50  * host-name
51  * sysctl
52  * user-homes

As you can see on line 10
"device-mapping-streetkid_VG_storage-streetkid_LV_home" has been started
and on line 43 "user-processes" has been started. The problem is that
"guix-daemon" hasn't been started (line 46) and "nscd" also hasn't been
started (line 47). Shouldn't Shepherd start these two services once
"user-processes" has been started?

I checked other services that depend on "user-processes" (see the attached
shepherd dependency graph for my system). The "mcron" service and "ntpd"
successfully start (lines 20 and 23). This still leaves the mystery of why
"guix-daemon" and "nscd" won't start even though "user-processes" HAS been
started.

I hope this makes things clearer.

Cheers
Markus

[-- Attachment #1.2: Type: text/html, Size: 3725 bytes --]

[-- Attachment #2: streetkid-system-config-shepherd-graph.gv --]
[-- Type: text/vnd.graphviz, Size: 8488 bytes --]

digraph "Guix shepherd-service" {
  "user-file-systems" [label = "user-file-systems", shape = box, fontname = sans];
  "file-systems" -> "user-file-systems" [color = darkviolet];
  "file-systems" [label = "file-systems", shape = box, fontname = sans];
  "user-processes" -> "file-systems" [color = dimgrey];
  "user-homes" -> "file-systems" [color = darkgoldenrod];
  "urandom-seed" -> "file-systems" [color = magenta];
  "user-processes" [label = "user-processes", shape = box, fontname = sans];
  "nscd" -> "user-processes" [color = darkseagreen];
  "guix-daemon" -> "user-processes" [color = dimgrey];
  "term-tty6" -> "user-processes" [color = peachpuff4];
  "term-tty5" -> "user-processes" [color = red];
  "term-tty4" -> "user-processes" [color = darkseagreen];
  "term-tty3" -> "user-processes" [color = darkgoldenrod];
  "term-tty2" -> "user-processes" [color = dimgrey];
  "term-tty1" -> "user-processes" [color = magenta];
  "term-console" -> "user-processes" [color = magenta];
  "syslogd" -> "user-processes" [color = darkseagreen];
  "ntpd" -> "user-processes" [color = red];
  "mcron" -> "user-processes" [color = darkgoldenrod];
  "nscd" [label = "nscd", shape = box, fontname = sans];
  "guix-daemon" [label = "guix-daemon", shape = box, fontname = sans];
  "term-tty6" [label = "term-tty6", shape = box, fontname = sans];
  "console-font-tty6" -> "term-tty6" [color = darkviolet];
  "console-font-tty6" [label = "console-font-tty6", shape = box, fontname = sans];
  "term-tty5" [label = "term-tty5", shape = box, fontname = sans];
  "console-font-tty5" -> "term-tty5" [color = peachpuff4];
  "console-font-tty5" [label = "console-font-tty5", shape = box, fontname = sans];
  "term-tty4" [label = "term-tty4", shape = box, fontname = sans];
  "console-font-tty4" -> "term-tty4" [color = red];
  "console-font-tty4" [label = "console-font-tty4", shape = box, fontname = sans];
  "term-tty3" [label = "term-tty3", shape = box, fontname = sans];
  "console-font-tty3" -> "term-tty3" [color = cyan3];
  "console-font-tty3" [label = "console-font-tty3", shape = box, fontname = sans];
  "term-tty2" [label = "term-tty2", shape = box, fontname = sans];
  "console-font-tty2" -> "term-tty2" [color = blue];
  "console-font-tty2" [label = "console-font-tty2", shape = box, fontname = sans];
  "term-tty1" [label = "term-tty1", shape = box, fontname = sans];
  "console-font-tty1" -> "term-tty1" [color = darkseagreen];
  "console-font-tty1" [label = "console-font-tty1", shape = box, fontname = sans];
  "term-console" [label = "term-console", shape = box, fontname = sans];
  "syslogd" [label = "syslogd", shape = box, fontname = sans];
  "term-console" -> "syslogd" [color = magenta];
  "ssh-daemon ssh sshd" -> "syslogd" [color = cyan3];
  "ssh-daemon ssh sshd" [label = "ssh-daemon ssh sshd", shape = box, fontname = sans];
  "ntpd" [label = "ntpd", shape = box, fontname = sans];
  "mcron" [label = "mcron", shape = box, fontname = sans];
  "user-homes" [label = "user-homes", shape = box, fontname = sans];
  "user-processes" -> "user-homes" [color = dimgrey];
  "urandom-seed" [label = "urandom-seed", shape = box, fontname = sans];
  "user-processes" -> "urandom-seed" [color = dimgrey];
  "root-file-system" [label = "root-file-system", shape = box, fontname = sans];
  "file-systems" -> "root-file-system" [color = darkviolet];
  "file-system-/home" -> "root-file-system" [color = darkseagreen];
  "file-system-/dev/pts" -> "root-file-system" [color = red];
  "file-system-/sys/kernel/debug" -> "root-file-system" [color = cyan3];
  "file-system-/dev/shm" -> "root-file-system" [color = peachpuff4];
  "file-system-/sys/firmware/efi/efivars" -> "root-file-system" [color = red];
  "file-system-/gnu/store" -> "root-file-system" [color = magenta];
  "udev" -> "root-file-system" [color = cyan3];
  "virtual-terminal" -> "root-file-system" [color = peachpuff4];
  "file-system-/home" [label = "file-system-/home", shape = box, fontname = sans];
  "file-systems" -> "file-system-/home" [color = darkviolet];
  "file-system-/dev/pts" [label = "file-system-/dev/pts", shape = box, fontname = sans];
  "file-systems" -> "file-system-/dev/pts" [color = darkviolet];
  "file-system-/sys/kernel/debug" [label = "file-system-/sys/kernel/debug", shape = box, fontname = sans];
  "file-systems" -> "file-system-/sys/kernel/debug" [color = darkviolet];
  "file-system-/dev/shm" [label = "file-system-/dev/shm", shape = box, fontname = sans];
  "file-systems" -> "file-system-/dev/shm" [color = darkviolet];
  "file-system-/sys/firmware/efi/efivars" [label = "file-system-/sys/firmware/efi/efivars", shape = box, fontname = sans];
  "file-systems" -> "file-system-/sys/firmware/efi/efivars" [color = darkviolet];
  "file-system-/gnu/store" [label = "file-system-/gnu/store", shape = box, fontname = sans];
  "file-systems" -> "file-system-/gnu/store" [color = darkviolet];
  "udev" [label = "udev", shape = box, fontname = sans];
  "device-mapping-streetkid_VG_storage-streetkid_LV_home" -> "udev" [color = blue];
  "file-system-/home" -> "udev" [color = darkseagreen];
  "file-system-/dev/pts" -> "udev" [color = red];
  "file-system-/sys/kernel/debug" -> "udev" [color = cyan3];
  "file-system-/dev/shm" -> "udev" [color = peachpuff4];
  "file-system-/sys/firmware/efi/efivars" -> "udev" [color = red];
  "file-system-/gnu/store" -> "udev" [color = magenta];
  "urandom-seed" -> "udev" [color = magenta];
  "term-tty6" -> "udev" [color = peachpuff4];
  "term-tty5" -> "udev" [color = red];
  "term-tty4" -> "udev" [color = darkseagreen];
  "term-tty3" -> "udev" [color = darkgoldenrod];
  "term-tty2" -> "udev" [color = dimgrey];
  "term-tty1" -> "udev" [color = magenta];
  "term-console" -> "udev" [color = magenta];
  "networking" -> "udev" [color = darkviolet];
  "device-mapping-streetkid_VG_storage-streetkid_LV_home" [label = "device-mapping-streetkid_VG_storage-streetkid_LV_home", shape = box, fontname = sans];
  "file-system-/home" -> "device-mapping-streetkid_VG_storage-streetkid_LV_home" [color = darkseagreen];
  "networking" [label = "networking", shape = box, fontname = sans];
  "rpc.nfsd" -> "networking" [color = darkseagreen];
  "ntpd" -> "networking" [color = red];
  "rpcbind-daemon" -> "networking" [color = dimgrey];
  "rpc.nfsd" [label = "rpc.nfsd", shape = box, fontname = sans];
  "nfs" -> "rpc.nfsd" [color = darkseagreen];
  "nfs" [label = "nfs", shape = box, fontname = sans];
  "rpcbind-daemon" [label = "rpcbind-daemon", shape = box, fontname = sans];
  "rpc.statd" -> "rpcbind-daemon" [color = cyan3];
  "nfs" -> "rpcbind-daemon" [color = darkseagreen];
  "idmap-daemon" -> "rpcbind-daemon" [color = dimgrey];
  "rpc.statd" [label = "rpc.statd", shape = box, fontname = sans];
  "rpc.mountd" -> "rpc.statd" [color = peachpuff4];
  "rpc.nfsd" -> "rpc.statd" [color = darkseagreen];
  "nfs" -> "rpc.statd" [color = darkseagreen];
  "rpc.mountd" [label = "rpc.mountd", shape = box, fontname = sans];
  "nfs" -> "rpc.mountd" [color = darkseagreen];
  "idmap-daemon" [label = "idmap-daemon", shape = box, fontname = sans];
  "virtual-terminal" [label = "virtual-terminal", shape = box, fontname = sans];
  "term-tty6" -> "virtual-terminal" [color = peachpuff4];
  "term-tty5" -> "virtual-terminal" [color = red];
  "term-tty4" -> "virtual-terminal" [color = darkseagreen];
  "term-tty3" -> "virtual-terminal" [color = darkgoldenrod];
  "term-tty2" -> "virtual-terminal" [color = dimgrey];
  "term-tty1" -> "virtual-terminal" [color = magenta];
  "host-name" [label = "host-name", shape = box, fontname = sans];
  "term-tty6" -> "host-name" [color = peachpuff4];
  "term-tty5" -> "host-name" [color = red];
  "term-tty4" -> "host-name" [color = darkseagreen];
  "term-tty3" -> "host-name" [color = darkgoldenrod];
  "term-tty2" -> "host-name" [color = dimgrey];
  "term-tty1" -> "host-name" [color = magenta];
  "term-console" -> "host-name" [color = magenta];
  "sysctl" [label = "sysctl", shape = box, fontname = sans];
  "loopback" [label = "loopback", shape = box, fontname = sans];
  "ssh-daemon ssh sshd" -> "loopback" [color = cyan3];
  "/proc/fs/nfsd" [label = "/proc/fs/nfsd", shape = box, fontname = sans];
  "rpc.statd" -> "/proc/fs/nfsd" [color = cyan3];
  "rpc.mountd" -> "/proc/fs/nfsd" [color = peachpuff4];
  "rpc.nfsd" -> "/proc/fs/nfsd" [color = darkseagreen];
  "nfs" -> "/proc/fs/nfsd" [color = darkseagreen];
  "rpc-pipefs" [label = "rpc-pipefs", shape = box, fontname = sans];
  "idmap-daemon" -> "rpc-pipefs" [color = dimgrey];

}