bug#25688: services: ssh-daemon quits too early

bug#25688: services: ssh-daemon quits too early

[ng0]

I have my openssh-service configured like this (not exactly, but it
serves an example well enough to not expose too much):

(service openssh-service-type
 (openssh-configuration
  (port-number 2200)
  (password-authentication? #t)
  (permit-root-login 'without-password)))

Now, when I reconfigure the system, ssh-daemon is started and I can ssh in.
When I reboot and try to ssh into the machine again, I see ssh-daemon is
stopped and I can't log in from a remote machine.

The issue seems to be that network-service is taking too long to come up
and therefore ssh-service is stopped at system boot similar to something
like this:

* starting ssh-daemon-service
* ssh-daemon depends on networking-service
* stopping ssh-daemon-service

some more services started, networking finally comes up while tor and
others have waited for networking to come up, but ssh-daemon is stopped
and doesn't wait for networking.

This system is currently on commit
883aab6462c49d4f4846a6f22168325e70227663

with no extra additions.
-- 
ng0 -- https://www.inventati.org/patternsinthechaos/

bug#25688: services: ssh-daemon quits too early

[Ludovic Courtès]

ng0 <contact.ng0@cryptolab.net> skribis:

> I have my openssh-service configured like this (not exactly, but it
> serves an example well enough to not expose too much):
>
> (service openssh-service-type
>  (openssh-configuration
>   (port-number 2200)
>   (password-authentication? #t)
>   (permit-root-login 'without-password)))
>
> Now, when I reconfigure the system, ssh-daemon is started and I can ssh in.
> When I reboot and try to ssh into the machine again, I see ssh-daemon is
> stopped and I can't log in from a remote machine.
>
> The issue seems to be that network-service is taking too long to come up
> and therefore ssh-service is stopped at system boot similar to something
> like this:
>
> * starting ssh-daemon-service
> * ssh-daemon depends on networking-service
> * stopping ssh-daemon-service

Could you send the relevant part of /var/log/shepherd.log, which shows
messages corresponding to the startup of the ‘networking’ and
‘ssh-daemon’ services?

TIA,
Ludo’.

bug#25688: services: ssh-daemon quits too early

[ng0]

On 17-02-11 15:40:38, Ludovic Courtès wrote:
> ng0 <contact.ng0@cryptolab.net> skribis:
> 
> > I have my openssh-service configured like this (not exactly, but it
> > serves an example well enough to not expose too much):
> >
> > (service openssh-service-type
> >  (openssh-configuration
> >   (port-number 2200)
> >   (password-authentication? #t)
> >   (permit-root-login 'without-password)))
> >
> > Now, when I reconfigure the system, ssh-daemon is started and I can ssh in.
> > When I reboot and try to ssh into the machine again, I see ssh-daemon is
> > stopped and I can't log in from a remote machine.
> >
> > The issue seems to be that network-service is taking too long to come up
> > and therefore ssh-service is stopped at system boot similar to something
> > like this:
> >
> > * starting ssh-daemon-service
> > * ssh-daemon depends on networking-service
> > * stopping ssh-daemon-service
> 
> Could you send the relevant part of /var/log/shepherd.log, which shows
> messages corresponding to the startup of the ‘networking’ and
> ‘ssh-daemon’ services?
> 
> TIA,
> Ludo’.

This is the ~120 lines of today. It reminded me of the real issue, which
is dbus could not be started, which very likely could be the reason for
the GNOME session crashing bug I reported.

2017-02-11 10:41:48 Service root has been started.
2017-02-11 10:41:50 starting services...
2017-02-11 10:41:50 Service root-file-system has been started.
2017-02-11 10:41:50 waiting for udevd...
2017-02-11 10:41:51 Service udev has been started.
2017-02-11 10:41:51 Service swap-/dev/sda3 has been started.
2017-02-11 10:41:51 Service user-file-systems has been started.
2017-02-11 10:41:53 Service file-system-/boot has been started.
2017-02-11 10:41:55 Service file-system-/media/pub has been started.
2017-02-11 10:41:55 Service file-system-/dev/pts has been started.
2017-02-11 10:41:55 Service file-system-/dev/shm has been started.
2017-02-11 10:41:55 Service file-system-/gnu/store has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/cpuset has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/cpu has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/cpuacct has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/memory has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/devices has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/freezer has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/blkio has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/perf_event has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/hugetlb has been started.
2017-02-11 10:41:55 Service file-system-/run/systemd has been started.
2017-02-11 10:41:55 Service file-system-/run/user has been started.
2017-02-11 10:41:55 Service file-system-/sys/fs/cgroup/elogind has been started.
2017-02-11 10:41:55 Service file-systems has been started.
2017-02-11 10:41:55 Service user-processes has been started.
2017-02-11 10:41:55 Service host-name has been started.
2017-02-11 10:41:55 Service user-homes could not be started.
2017-02-11 10:41:56 Service nscd has been started.
2017-02-11 10:41:56 Service guix-daemon has been started.
2017-02-11 10:41:57 Service urandom-seed has been started.
2017-02-11 10:41:57 Service syslogd has been started.
2017-02-11 10:41:58 Service loopback has been started.
2017-02-11 10:41:58 Service term-tty6 has been started.
2017-02-11 10:41:58 Service term-tty5 has been started.
2017-02-11 10:41:58 Service term-tty4 has been started.
2017-02-11 10:41:58 Service term-tty3 has been started.
2017-02-11 10:41:58 Service term-tty2 has been started.
2017-02-11 10:41:58 Service term-tty1 has been started.
2017-02-11 10:42:04 Service console-font-tty1 has been started.
2017-02-11 10:42:05 Service console-font-tty2 has been started.
2017-02-11 10:42:05 Service console-font-tty3 has been started.
2017-02-11 10:42:05 Service console-font-tty4 has been started.
2017-02-11 10:42:05 Service console-font-tty5 has been started.
2017-02-11 10:42:05 Service console-font-tty6 has been started.
2017-02-11 10:42:10 Service dbus-system could not be started.
2017-02-11 10:42:10 Service networking depends on dbus-system.
2017-02-11 10:42:10 Service networking could not be started.
2017-02-11 10:42:10 Service ssh-daemon depends on networking.
2017-02-11 10:42:10 Service ssh-daemon could not be started.
2017-02-11 10:42:11 Service console-keymap has been started.
2017-02-11 10:42:11 Service tor has been started.
2017-02-11 10:42:12 Service dbus-system has been started.
2017-02-11 10:42:12 Service upower-daemon has been started.
2017-02-11 10:42:12 Service networking has been started.
2017-02-11 10:42:13 Service avahi-daemon has been started.
2017-02-11 10:42:13 Service display-manager has been started.
2017-02-11 13:50:08 Respawning term-tty2.
2017-02-11 13:50:08 Service term-tty2 has been started.
2017-02-11 13:51:21 Service ssh-daemon has been started.
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@2017-02-11 13:55:25 Service root has been started.
2017-02-11 13:55:27 starting services...
2017-02-11 13:55:27 Service root-file-system has been started.
2017-02-11 13:55:27 waiting for udevd...
2017-02-11 13:55:27 Service udev has been started.
2017-02-11 13:55:27 Service swap-/dev/sda3 has been started.
2017-02-11 13:55:27 Service user-file-systems has been started.
2017-02-11 13:55:30 Service file-system-/boot has been started.
2017-02-11 13:55:31 Service file-system-/media/pub has been started.
2017-02-11 13:55:31 Service file-system-/dev/pts has been started.
2017-02-11 13:55:31 Service file-system-/dev/shm has been started.
2017-02-11 13:55:31 Service file-system-/gnu/store has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/cpuset has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/cpu has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/cpuacct has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/memory has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/devices has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/freezer has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/blkio has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/perf_event has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/hugetlb has been started.
2017-02-11 13:55:32 Service file-system-/run/systemd has been started.
2017-02-11 13:55:32 Service file-system-/run/user has been started.
2017-02-11 13:55:32 Service file-system-/sys/fs/cgroup/elogind has been started.
2017-02-11 13:55:32 Service file-systems has been started.
2017-02-11 13:55:32 Service user-processes has been started.
2017-02-11 13:55:32 Service host-name has been started.
2017-02-11 13:55:32 Service user-homes could not be started.
2017-02-11 13:55:33 Service nscd has been started.
2017-02-11 13:55:33 Service guix-daemon has been started.
2017-02-11 13:55:33 Service urandom-seed has been started.
2017-02-11 13:55:34 Service syslogd has been started.
2017-02-11 13:55:34 Service loopback has been started.
2017-02-11 13:55:34 Service term-tty6 has been started.
2017-02-11 13:55:34 Service term-tty5 has been started.
2017-02-11 13:55:34 Service term-tty4 has been started.
2017-02-11 13:55:34 Service term-tty3 has been started.
2017-02-11 13:55:34 Service term-tty2 has been started.
2017-02-11 13:55:34 Service term-tty1 has been started.
2017-02-11 13:55:39 Service console-font-tty1 has been started.
2017-02-11 13:55:41 Service console-font-tty2 has been started.
2017-02-11 13:55:41 Service console-font-tty3 has been started.
2017-02-11 13:55:41 Service console-font-tty4 has been started.
2017-02-11 13:55:41 Service console-font-tty5 has been started.
2017-02-11 13:55:41 Service console-font-tty6 has been started.
2017-02-11 13:55:46 Service dbus-system could not be started.
2017-02-11 13:55:46 Service networking depends on dbus-system.
2017-02-11 13:55:46 Service networking could not be started.
2017-02-11 13:55:46 Service ssh-daemon depends on networking.
2017-02-11 13:55:46 Service ssh-daemon could not be started.
2017-02-11 13:55:46 Service console-keymap has been started.
2017-02-11 13:55:46 Service tor has been started.
2017-02-11 13:55:47 Service dbus-system has been started.
2017-02-11 13:55:47 Service upower-daemon has been started.
2017-02-11 13:55:47 Service networking has been started.
2017-02-11 13:55:49 Service avahi-daemon has been started.
2017-02-11 13:55:49 Service display-manager has been started.
2017-02-11 14:02:47 Service ssh-daemon is not running.
2017-02-11 14:02:48 Service ssh-daemon has been started.

The last 2 lines above is when I issued locally 'sudo herd restart
ssh-daemon'.



-- 
ng0 -- https://www.inventati.org/patternsinthechaos/

bug#25688: services: ssh-daemon quits too early

[Ludovic Courtès]

ng0 <contact.ng0@cryptolab.net> skribis:

> 2017-02-11 13:55:46 Service dbus-system could not be started.
> 2017-02-11 13:55:46 Service networking depends on dbus-system.
> 2017-02-11 13:55:46 Service networking could not be started.
> 2017-02-11 13:55:46 Service ssh-daemon depends on networking.
> 2017-02-11 13:55:46 Service ssh-daemon could not be started.

Looks like the root cause is the ‘dbus-system’ service (dbus-daemon)
failed to start.  Could you check if there are relevant messages from
dbus-daemon in /var/log/messages or in the console?

TIA!

Ludo’.

bug#25688: services: ssh-daemon quits too early

[ng0]

Fixed by closing a bug where (locale-libcs) was blocking the locale
generation.

On 17-02-11 22:18:08, Ludovic Courtès wrote:
> ng0 <contact.ng0@cryptolab.net> skribis:
> 
> > 2017-02-11 13:55:46 Service dbus-system could not be started.
> > 2017-02-11 13:55:46 Service networking depends on dbus-system.
> > 2017-02-11 13:55:46 Service networking could not be started.
> > 2017-02-11 13:55:46 Service ssh-daemon depends on networking.
> > 2017-02-11 13:55:46 Service ssh-daemon could not be started.
> 
> Looks like the root cause is the ‘dbus-system’ service (dbus-daemon)
> failed to start.  Could you check if there are relevant messages from
> dbus-daemon in /var/log/messages or in the console?
> 
> TIA!
> 
> Ludo’.

-- 
ng0 -- https://www.inventati.org/patternsinthechaos/

bug#25688: services: ssh-daemon quits too early

[Ludovic Courtès]

ng0 <contact.ng0@cryptolab.net> skribis:

> Fixed by closing a bug where (locale-libcs) was blocking the locale
> generation.

Uh, weird.

Ludo'.

bug#25688: services: ssh-daemon quits too early

[ng0]

On 17-02-12 17:17:04, Ludovic Courtès wrote:
> ng0 <contact.ng0@cryptolab.net> skribis:
> 
> > Fixed by closing a bug where (locale-libcs) was blocking the locale
> > generation.
> 
> Uh, weird.
> 
> Ludo'.

Definitely... but let's just assume it's a unique problem until someone
else comes up with the same problem.
-- 
ng0 -- https://www.inventati.org/patternsinthechaos/

bug#25688: services: ssh-daemon quits too early

[Leo Famulari]

On Sat, Feb 11, 2017 at 03:00:13PM +0000, ng0 wrote:
> This is the ~120 lines of today. It reminded me of the real issue, which
> is dbus could not be started, which very likely could be the reason for
> the GNOME session crashing bug I reported.

Are you still having this problem (dbus-system could not be started)?

bug#25688: services: ssh-daemon quits too early

[ng0]

On 17-03-02 16:58:35, Leo Famulari wrote:
> On Sat, Feb 11, 2017 at 03:00:13PM +0000, ng0 wrote:
> > This is the ~120 lines of today. It reminded me of the real issue, which
> > is dbus could not be started, which very likely could be the reason for
> > the GNOME session crashing bug I reported.
> 
> Are you still having this problem (dbus-system could not be started)?

I've recently started to d othe system from 0 again to rule out
offloading issues, but ssh-daemon started failing again. I'll look into
the issue on the weekend.

bug#25688: services: ssh-daemon quits too early

[ng0]

On 17-03-02 22:20:44, ng0 wrote:
> On 17-03-02 16:58:35, Leo Famulari wrote:
> > On Sat, Feb 11, 2017 at 03:00:13PM +0000, ng0 wrote:
> > > This is the ~120 lines of today. It reminded me of the real issue, which
> > > is dbus could not be started, which very likely could be the reason for
> > > the GNOME session crashing bug I reported.
> > 
> > Are you still having this problem (dbus-system could not be started)?
> 
> I've recently started to d othe system from 0 again to rule out
> offloading issues, but ssh-daemon started failing again. I'll look into
> the issue on the weekend.
> 
> 
> 
I can confirm this continues to be a problem on the new system,
repeatedly:

Service user-homes could not be started.
The same for ssh-daemon, but now dbus-system can be started and the
problem could therefore be user-homes.

bug#25688: services: ssh-daemon quits too early

[ng0]

On 17-03-03 11:36:12, ng0 wrote:
> On 17-03-02 22:20:44, ng0 wrote:
> > On 17-03-02 16:58:35, Leo Famulari wrote:
> > > On Sat, Feb 11, 2017 at 03:00:13PM +0000, ng0 wrote:
> > > > This is the ~120 lines of today. It reminded me of the real issue, which
> > > > is dbus could not be started, which very likely could be the reason for
> > > > the GNOME session crashing bug I reported.
> > > 
> > > Are you still having this problem (dbus-system could not be started)?
> > 
> > I've recently started to d othe system from 0 again to rule out
> > offloading issues, but ssh-daemon started failing again. I'll look into
> > the issue on the weekend.
> > 
> > 
> > 
> I can confirm this continues to be a problem on the new system,
> repeatedly:
> 
> Service user-homes could not be started.
> The same for ssh-daemon, but now dbus-system can be started and the
> problem could therefore be user-homes.
> 
> 
> 

Correction, and I'm afraid I have no further insights:

With my system config now moving towards a working all-purpose server
this has been fixed.
On the old (desktop system) configuration this problem continued to
exist, and will probably still be reproducible.