syslogd can't be started when running Guix in Docker

syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Hi everyone,

I am exploring the possibility of running Guix under macOS via Docker.
First I ran 'guix system docker-image` on a minimal system
configuration, which I then transferred to the Mac and started.
Overall it works admirably well. There are however a few services
that don't work, in particulary syslogd:

   root@guix /# herd status
   Started:
    + file-systems
    + guix-daemon
    + host-name
    + loopback
    + root
    + root-file-system
    + udev
    + urandom-seed
    + user-file-systems
    + user-processes
    + virtual-terminal
   Stopped:
    - nscd
    - ssh-daemon
    - syslogd
   One-shot:
    * user-homes

   root@guix /# herd start syslogd
   Service syslogd could not be started.
   herd: failed to start service syslogd

I have no idea why this should fail, and worse, I have no idea how to
debug this. I checked /var/log/messages and /var/log/debug, but I can't
see anything looking like an error message except for shepherd reporting
that syslogs and ssh-daemon could not be started.

I'd be grateful for ideas and suggestions on how to figure out what is
going wrong. I'd also be interested in working dockerized versions
of Guix for comparison.

Thanks in advance,
  Konrad

Re: syslogd can't be started when running Guix in Docker

[Maxim Cournoyer]

Hi Konrad,

On July 2, 2019 5:16:40 PM UTC, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
>Hi everyone,
>
>I am exploring the possibility of running Guix under macOS via Docker.
>First I ran 'guix system docker-image` on a minimal system
>configuration, which I then transferred to the Mac and started.
>Overall it works admirably well. There are however a few services
>that don't work, in particulary syslogd:

Try running the Docker container in privileged mode (docker run --privileged); I think this will allow the necessary Guix  services dependencies of the syslog service to run.

I forgot the details, bit I think it has to do with syslog depending on something you'd have on a real system but not in a container.


>   root@guix /# herd status
>   Started:
>    + file-systems
>    + guix-daemon
>    + host-name
>    + loopback
>    + root
>    + root-file-system
>    + udev
>    + urandom-seed
>    + user-file-systems
>    + user-processes
>    + virtual-terminal
>   Stopped:
>    - nscd
>    - ssh-daemon
>    - syslogd
>   One-shot:
>    * user-homes
>
>   root@guix /# herd start syslogd
>   Service syslogd could not be started.
>   herd: failed to start service syslogd
>
>I have no idea why this should fail, and worse, I have no idea how to
>debug this. 

I think I've used the emacs-guix package to see the dependency graphs of services and that helped me IIRC. Plus peaking in the services source code.

HTH,

Maxim

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Hi Maxim,

Thanks for your reply!

> Try running the Docker container in privileged mode (docker run
> --privileged); I think this will allow the necessary Guix services
> dependencies of the syslog service to run.

I am already running in priviledged mode. Without it, I couldn't
build packages.

> I think I've used the emacs-guix package to see the dependency graphs
> of services and that helped me IIRC. Plus peaking in the services
> source code.

I tried that but I am not enough of a low-level Linux expert to get
much benefit from it.

What I did notice is that syslogd is actually running once it has been
started. It shows up in 'ps', and /var/log/messages keeps accumulating
entries. /var/run/syslog.pid exists and contains the pid of the running
syslogd process.

All this suggests that perhaps the problem is not with syslogd, but with
shepherd wrongly concluding that syslogd failed.

Looks like I will have to dig a bit more into shepherd :-(

Konrad.

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

On 03/07/2019 10:12, Konrad Hinsen wrote:

> All this suggests that perhaps the problem is not with syslogd, but with
> shepherd wrongly concluding that syslogd failed.
>
> Looks like I will have to dig a bit more into shepherd :-(

My hypothesis turned out to be correct: increasing the timeout was 
sufficient to get syslogd working.

Konrad.

Re: syslogd can't be started when running Guix in Docker

[Maxim Cournoyer]

Hi Konrad!

I'm happy to read you could successfully make it work! It's strange that you had that issue though!

Cheers,

Maxim

On July 3, 2019 11:17:03 AM UTC, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
>On 03/07/2019 10:12, Konrad Hinsen wrote:
>
>> All this suggests that perhaps the problem is not with syslogd, but
>with
>> shepherd wrongly concluding that syslogd failed.
>>
>> Looks like I will have to dig a bit more into shepherd :-(
>
>My hypothesis turned out to be correct: increasing the timeout was 
>sufficient to get syslogd working.
>
>Konrad.

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> I'm happy to read you could successfully make it work! It's strange
> that you had that issue though!

By now I have a system that starts all services without failure. I got
there by increasing the timeout to 30 s everywhere (compared to the
default value of 5). Perhaps 30 is more than necessary, I didn't try
intermediate values. So it looks like Guix in Docker is a bit slow to
boot, at least on the Mac.

A more fundamental question is whether I will be able to do useful work
with this setup. It is well known that host file access from Docker on
macOS is slow, but I didn't realize how slow it is until I compiled Guix
in... four hours. On the same machine booted to Linux, it's about 40
minutes.

Konrad.

Re: syslogd can't be started when running Guix in Docker

[Maxim Cournoyer]

Hello!

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> I'm happy to read you could successfully make it work! It's strange
>> that you had that issue though!
>
> By now I have a system that starts all services without failure. I got
> there by increasing the timeout to 30 s everywhere (compared to the
> default value of 5). Perhaps 30 is more than necessary, I didn't try
> intermediate values. So it looks like Guix in Docker is a bit slow to
> boot, at least on the Mac.

I see! The unusual slowness would explain why the default timeout needed
to be relaxed.

> A more fundamental question is whether I will be able to do useful work
> with this setup. It is well known that host file access from Docker on
> macOS is slow, but I didn't realize how slow it is until I compiled Guix
> in... four hours. On the same machine booted to Linux, it's about 40
> minutes.

It's an interesting experiment for sure :-) Please let us know of your
findings!

Maxim

Re: syslogd can't be started when running Guix in Docker

[Ludovic Courtès]

Hi,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> I'm happy to read you could successfully make it work! It's strange
>> that you had that issue though!
>
> By now I have a system that starts all services without failure. I got
> there by increasing the timeout to 30 s everywhere (compared to the
> default value of 5). Perhaps 30 is more than necessary, I didn't try
> intermediate values. So it looks like Guix in Docker is a bit slow to
> boot, at least on the Mac.
>
> A more fundamental question is whether I will be able to do useful work
> with this setup. It is well known that host file access from Docker on
> macOS is slow, but I didn't realize how slow it is until I compiled Guix
> in... four hours. On the same machine booted to Linux, it's about 40
> minutes.

Ouch.  Is this a fact of life or are there special settings that could
make it less slow?

Five seconds to spawn a process is already relatively long; we could
increase it a little, but the slowness here is a concern.

Thanks for sharing your experience!

Ludo’.

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Hi Ludo,

>> A more fundamental question is whether I will be able to do useful work
>> with this setup. It is well known that host file access from Docker on
>> macOS is slow, but I didn't realize how slow it is until I compiled Guix
>> in... four hours. On the same machine booted to Linux, it's about 40
>> minutes.
>
> Ouch.  Is this a fact of life or are there special settings that could
> make it less slow?

Docker for macOS has some options that use caching for more efficient
host filesystem access, but they come with restrictions, in particular
file changes on one side not being propagated immediately to the other
side.

> Five seconds to spawn a process is already relatively long; we could
> increase it a little, but the slowness here is a concern.

Whereas the problems with file access are known and understood
(ultimately it is the important differences in low-level file handling
between Linux and macOS), I have no idea why the startup time for these
daemons is so high. There is no significant CPU performance overhead
when running Guix in Docker, nor do the daemons require any access to
files managed by macOS.

Konrad.

Re: syslogd can't be started when running Guix in Docker

[Ludovic Courtès]

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> Five seconds to spawn a process is already relatively long; we could
>> increase it a little, but the slowness here is a concern.
>
> Whereas the problems with file access are known and understood
> (ultimately it is the important differences in low-level file handling
> between Linux and macOS), I have no idea why the startup time for these
> daemons is so high. There is no significant CPU performance overhead
> when running Guix in Docker, nor do the daemons require any access to
> files managed by macOS.

Do you observe similar slowness with other Docker images—e.g., Debian
images?

Thanks,
Ludo’.

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Ludovic Courtès <ludo@gnu.org> writes:

> Do you observe similar slowness with other Docker images—e.g., Debian
> images?

No. But I don't observe any slowness with Guix either. Once it's up and
running, performance is normal (except for the known problem with access
to macOS file systems). It's only the daemon startup failure that
suggests a problem. I haven't had such problems with other Docker images
based on Debian or Ubuntu.

Konrad.

Re: syslogd can't be started when running Guix in Docker

[Ludovic Courtès]

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Do you observe similar slowness with other Docker images—e.g., Debian
>> images?
>
> No. But I don't observe any slowness with Guix either. Once it's up and
> running, performance is normal (except for the known problem with access
> to macOS file systems). It's only the daemon startup failure that
> suggests a problem. I haven't had such problems with other Docker images
> based on Debian or Ubuntu.

Hmm, it’s hard to draw any conclusion though, because systemd can do
things concurrently, so perhaps the process startup slowness is
invisible because it’s able to perform other startup work while waiting
for daemons.  (It also spawns fewer daemons because some of these are
subsumed by PID 1.)

Who knows!

Ludo’.