Re: Services and log management/monitoring

[rendaw <7e9wc56emjakcm@s.rendaw.me>]

On 4/18/19 6:20 AM, Ludovic Courtès wrote:
> Hello!
>
> The kind of “service” that Guix manages is those described here:
>
>   https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html
>
> The first paragraph tries to explain it:
>
>   Here we define a “service” as, broadly, something that extends the
>   functionality of the operating system.  Often a service is a process—a
>   “daemon”—started when the system boots: a secure shell server, a Web
>   server, the Guix build daemon, etc.  Sometimes a service is a daemon
>   whose execution can be triggered by another daemon—e.g., an FTP server
>   started by ‘inetd’ or a D-Bus service activated by ‘dbus-daemon’.
>   Occasionally, a service does not map to a daemon.  For instance, the
>   “account” service collects user accounts and makes sure they exist when
>   the system runs; the “udev” service collects device management rules and
>   makes them available to the eudev daemon; the ‘/etc’ service populates
>   the ‘/etc’ directory of the system.
>
> For services that “map to a daemon”, you’d extend
> ‘shepherd-root-service-type’ by providing a Shepherd service.  A
> Shepherd service is a service managed by PID 1.  You can list them on a
> running system by running ‘herd status’ as root.
>
> IOW, Shepherd services are a special case of service.  Just like D-Bus
> services are another special case, etc.
>
> About logging: Shepherd does very little in that area.  It limits itself
> to providing a #:log-file parameter to capture the processes’s
> stdout/stderr to a file.  Other than that we usually configure daemons
> to write to syslog, which provides more flexibility regarding storage
> and filtering of log entries.
Okay, this makes sense.  I guess if I need filtering/searching I should
probably look at packaging syslog-ng.  IIRC all Shepherd services log to
syslog, but do non-Shepherd services log somewhere?
>
> A “non-Shepherd service” as you call them doesn’t necessarily map to a
> process, so there’s potentially nothing to log in the first place.

It looks like Shepherd services don't necessarily map to a process
either - on this graph:

https://www.gnu.org/software/guix/manual/en/html_node/Shepherd-Services.html#Shepherd-Services

various file systems, host-name, etc are also shown in the graph.

>
> Does that answer your questions?
>
> Thanks,
> Ludo’.

I think fundamentally what I'd most like to know is when should I use a
Shepherd service vs a non-Shepherd service.  Maybe it's as simple as: if
you don't have any specific requirements always define a Shepherd service.

It looks like both Shepherd services and non-Shepherd services:

* Can have dependencies
* Might not necessarily be a process
* Can run an arbitrary expression

which doesn't give me direction for why I should use one rather than the
other.

What might be confusing me is the documentation seems to imply a
dichotomy of "Shepherd services" and "everything else", when perhaps
there's actually a large number of service types in a rather flat hierarchy:

* Shepherd services: start automatically, can be disabled, have separate
start + stop methods
* inet services: start on network request
* mcron services: start at time
* d-bus services: ??
* etc

and Shepherd services are just one of many types of service
differentiated by start conditions? This is speculation of course.

In older distros I think perhaps the differentiation was more simple:
services are processes started/stopped by run levels, and everything
else was not a service.  So services were like X11, inetd, and then
there were non-service things like fstab, cron entries, inetd "services"
configured in your inetd config file, etc.

In systemd everything's a "unit" which can start/stop a process and form
a graph, but then you have specialized hook units like "sockets" and
"timers" that will control other units which are outside the graph (and
thus won't be started based on graph order).  The whole graph is
executed when the system starts.

As a metaphor, maybe Shepherd services would be similar to the plain
units in systemd.

Beyond what a service actually is though I have a few more questions:

* Both
https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html
and
https://www.gnu.org/software/guix/manual/en/html_node/Shepherd-Services.html#Shepherd-Services
appear to show a dependency graph.  Are the dependency graphs Shepherd
and non-Shepherd services entirely separate?  Or maybe I'm completely
misunderstanding "extension" in this context.  Can an inet service
depend on a non-inet service?  Can an inet service depend on a d-bus
service?
* Is there a way to hook into service events - that is, run some code
when a service starts or stops?