Re: shepherd logging; console-agetty-service

[Danny Milosavljevic <dannym@scratchpost.org>]

Hi Ludo,

On Sat, 17 Feb 2018 17:20:06 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Actually, could [shepherd] use syslog(3), which writes to /dev/log?

As far as I understand the meaning overload in Linux, there's these:

* syslog(2) - accesses the kernel log ringbuffer directly.
glibc (rightly) pretends this doesn't exist.
* /dev/kmsg - userspace access to read and write to the kernel
log ringbuffer.  It doesn't really work like a regular file.
If you write(2) to it, that gets interpreted as a *line*
of max length (1024 - 32) and this line gets added to the
ringbuffer as a new message.
Reading is weird too - but we won't use that directly.
* syslog(3) - actually writes to /dev/log, a socket, which is
usually listened to by a user-space process like syslogd.
No kernel involved anywhere (well, I guess the BSD socket
layer is involved...).

If shepherd used /dev/log in the beginning that would mean that
syslogd would have to be running before shepherd starts up.

So syslogd would have pid 1, I guess ;)

If we instead used /dev/kmsg, there would not really be a hard dependency.
Linux takes care that the messages are printed to the console
until a syslogd (maybe) appears, at which point syslogd will
log the already-printed (to /dev/kmsg) messages to its output files.

syslogd also fetches LATER messages from /dev/kmsg and
appends them to its output files likewise as time goes by.

So the UNIXy good-enough way would be for us to only use /dev/kmsg, the end :)

The complete way (which would avoid a hypothetical problem of overloading
the ringbuffer with messages so we would lose some of them) would be
to have shepherd start out using /dev/kmsg and switch over to using /dev/log
when it's there (maybe use inotify and watch until /dev/log appears).
Switch back to /dev/kmsg if /dev/log breaks again.

Also possible would be to have shepherd provide /dev/log to user services
and have shepherd proxy them to /dev/kmsg - that sounds like a real easy way
to get into a cycle if we aren't careful - although it would be conceptually
nicer.

In fact, the Linux kernel could have done that transparently.  They probably
just didn't like the failover handling involved :)

Also, I checked glibc misc/syslog.c vsyslog and they fall back to
/dev/console - disappointing.
They could have fallen back to /dev/kmsg ...

But then if syslogd crashed nobody would re-enable console logging and it wouldn't
be seen until syslogd started up again...  it's actually starting to get
complicated again :)

> > Also a way of capturing stderr and stdout (and maybe even /dev/log) of services
> > would be nice.  
> 
> Yes.  Though again capturing service stdout/stderr is kinda redundant
> with what syslogd does.  What I like in journald is the fact that it
> unifies all logging facilities, and also connects them to service
> management.

Yeah, its nice nowadays after one gets used to it.

In the end, it depends on whether one wants a system overview or a
service-by-service overview.

Older UNIX had lots of small programs working together in harmony to
achieve one task that the user cared about.

If you just watched the logs of one of them that wouldn't really help
that much because you'd not see the parts the other guys did.

And if you then watched *their* logs in an extra terminal or something
you've have to manually cobble together the timeline in your head to
make sense of what you are seeing.

Sometimes some of the services involved were on other machines.

So it was understandable that syslogd came to be a "global" thing
(sometimes on a central server only).

Nowadays with all the huge-ish services like dnsmasq which do
everything on their own (DNS, DHCP, TFTP, kitchensink) just checking
its log is enough.

Not sure how (or whether) to unify both use cases.

> Right, so we’d need shepherd to filter these and pass them through
> syslog(3), which ensures correct message formatting.  That’s what
> <https://cgit.freedesktop.org/systemd/systemd/tree/src/journal/journald-kmsg.c>
> does apparently (thanks for the link!).

Yeah.