Services and log management/monitoring

Services and log management/monitoring

[rendaw]

The manual talks about standard services vs shepherd services, but
doesn't really discuss which either are.  My impression is a standard
service is simply a process guix runs on the system (at startup, based
on the graph?), where a shepherd service is a process that is managed
(may be manually started/stopped, automatically restarted, is watched)
by shepherd.

Is that interpretation correct?  My expectation would then be that you
should generally use shepherd services, except when you want very fine
control over how the service is managed.

Do shepherd services automatically get enabled and start when configured
in a guix config?

Okay, just reread everything and I'm quite confused.  It says shepherd
is pid 1, so "shepherd" services are the more fundamental service type?
But the manual says that shepherd services are a particular type of
service, which implies the opposite.  Why would one use "service"
instead of "shepherd-service" (or vice-versa)?

Is it possible to define a service which depends on a shepherd-service?
What about the reverse?

--

On logging - one of the things I like about systemd is that all services
log their stdout/err and there's a standard way to retrieve logs, and
especially query logs from a particular time range/boot.  It also
exposes dbus events when service statuses change - so it's pretty easy
to send a notification if a service dies for instance, maybe with the
logs from the 10m leading up to the issue.

I couldn't find anything about logging and monitoring in the guix manual
(other than statistic collecting agents/dashboards).  In the shepherd
manual all I could find was a note that services send stdout/stderr to
syslog.  Are any of the above possible with this setup?  I've used
syslog systems for years up until systemd, but my impression was that it
was basically a pile of random files you can read manually if you really
must.

Do non-shepherd services also log somewhere?

Re: Services and log management/monitoring

[Ludovic Courtès]

Hello!

The kind of “service” that Guix manages is those described here:

  https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html

The first paragraph tries to explain it:

  Here we define a “service” as, broadly, something that extends the
  functionality of the operating system.  Often a service is a process—a
  “daemon”—started when the system boots: a secure shell server, a Web
  server, the Guix build daemon, etc.  Sometimes a service is a daemon
  whose execution can be triggered by another daemon—e.g., an FTP server
  started by ‘inetd’ or a D-Bus service activated by ‘dbus-daemon’.
  Occasionally, a service does not map to a daemon.  For instance, the
  “account” service collects user accounts and makes sure they exist when
  the system runs; the “udev” service collects device management rules and
  makes them available to the eudev daemon; the ‘/etc’ service populates
  the ‘/etc’ directory of the system.

For services that “map to a daemon”, you’d extend
‘shepherd-root-service-type’ by providing a Shepherd service.  A
Shepherd service is a service managed by PID 1.  You can list them on a
running system by running ‘herd status’ as root.

IOW, Shepherd services are a special case of service.  Just like D-Bus
services are another special case, etc.

About logging: Shepherd does very little in that area.  It limits itself
to providing a #:log-file parameter to capture the processes’s
stdout/stderr to a file.  Other than that we usually configure daemons
to write to syslog, which provides more flexibility regarding storage
and filtering of log entries.

A “non-Shepherd service” as you call them doesn’t necessarily map to a
process, so there’s potentially nothing to log in the first place.

Does that answer your questions?

Thanks,
Ludo’.

Re: Services and log management/monitoring

[rendaw]

On 4/18/19 6:20 AM, Ludovic Courtès wrote:
> Hello!
>
> The kind of “service” that Guix manages is those described here:
>
>   https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html
>
> The first paragraph tries to explain it:
>
>   Here we define a “service” as, broadly, something that extends the
>   functionality of the operating system.  Often a service is a process—a
>   “daemon”—started when the system boots: a secure shell server, a Web
>   server, the Guix build daemon, etc.  Sometimes a service is a daemon
>   whose execution can be triggered by another daemon—e.g., an FTP server
>   started by ‘inetd’ or a D-Bus service activated by ‘dbus-daemon’.
>   Occasionally, a service does not map to a daemon.  For instance, the
>   “account” service collects user accounts and makes sure they exist when
>   the system runs; the “udev” service collects device management rules and
>   makes them available to the eudev daemon; the ‘/etc’ service populates
>   the ‘/etc’ directory of the system.
>
> For services that “map to a daemon”, you’d extend
> ‘shepherd-root-service-type’ by providing a Shepherd service.  A
> Shepherd service is a service managed by PID 1.  You can list them on a
> running system by running ‘herd status’ as root.
>
> IOW, Shepherd services are a special case of service.  Just like D-Bus
> services are another special case, etc.
>
> About logging: Shepherd does very little in that area.  It limits itself
> to providing a #:log-file parameter to capture the processes’s
> stdout/stderr to a file.  Other than that we usually configure daemons
> to write to syslog, which provides more flexibility regarding storage
> and filtering of log entries.
Okay, this makes sense.  I guess if I need filtering/searching I should
probably look at packaging syslog-ng.  IIRC all Shepherd services log to
syslog, but do non-Shepherd services log somewhere?
>
> A “non-Shepherd service” as you call them doesn’t necessarily map to a
> process, so there’s potentially nothing to log in the first place.

It looks like Shepherd services don't necessarily map to a process
either - on this graph:

https://www.gnu.org/software/guix/manual/en/html_node/Shepherd-Services.html#Shepherd-Services

various file systems, host-name, etc are also shown in the graph.

>
> Does that answer your questions?
>
> Thanks,
> Ludo’.

I think fundamentally what I'd most like to know is when should I use a
Shepherd service vs a non-Shepherd service.  Maybe it's as simple as: if
you don't have any specific requirements always define a Shepherd service.

It looks like both Shepherd services and non-Shepherd services:

* Can have dependencies
* Might not necessarily be a process
* Can run an arbitrary expression

which doesn't give me direction for why I should use one rather than the
other.

What might be confusing me is the documentation seems to imply a
dichotomy of "Shepherd services" and "everything else", when perhaps
there's actually a large number of service types in a rather flat hierarchy:

* Shepherd services: start automatically, can be disabled, have separate
start + stop methods
* inet services: start on network request
* mcron services: start at time
* d-bus services: ??
* etc

and Shepherd services are just one of many types of service
differentiated by start conditions? This is speculation of course.

In older distros I think perhaps the differentiation was more simple:
services are processes started/stopped by run levels, and everything
else was not a service.  So services were like X11, inetd, and then
there were non-service things like fstab, cron entries, inetd "services"
configured in your inetd config file, etc.

In systemd everything's a "unit" which can start/stop a process and form
a graph, but then you have specialized hook units like "sockets" and
"timers" that will control other units which are outside the graph (and
thus won't be started based on graph order).  The whole graph is
executed when the system starts.

As a metaphor, maybe Shepherd services would be similar to the plain
units in systemd.

Beyond what a service actually is though I have a few more questions:

* Both
https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html
and
https://www.gnu.org/software/guix/manual/en/html_node/Shepherd-Services.html#Shepherd-Services
appear to show a dependency graph.  Are the dependency graphs Shepherd
and non-Shepherd services entirely separate?  Or maybe I'm completely
misunderstanding "extension" in this context.  Can an inet service
depend on a non-inet service?  Can an inet service depend on a d-bus
service?
* Is there a way to hook into service events - that is, run some code
when a service starts or stops?

Re: Services and log management/monitoring

[Ludovic Courtès]

Hi,

rendaw <7e9wc56emjakcm@s.rendaw.me> skribis:

> I think fundamentally what I'd most like to know is when should I use a
> Shepherd service vs a non-Shepherd service.  Maybe it's as simple as: if
> you don't have any specific requirements always define a Shepherd service.

It’s hard to answer that question in the abstract.  Do you have an
example in mind that we could work through?

> As a metaphor, maybe Shepherd services would be similar to the plain
> units in systemd.

Yes, I think so.

> Beyond what a service actually is though I have a few more questions:
>
> * Both
> https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html
> and
> https://www.gnu.org/software/guix/manual/en/html_node/Shepherd-Services.html#Shepherd-Services
> appear to show a dependency graph.

The first page shows a service extension graph.

The second page shows a graph of dependencies among Shepherd services.

These are two different beasts.

>   Are the dependency graphs Shepherd and non-Shepherd services
> entirely separate?  Or maybe I'm completely misunderstanding
> "extension" in this context.  Can an inet service depend on a non-inet
> service?  Can an inet service depend on a d-bus service?  * Is there a
> way to hook into service events - that is, run some code when a
> service starts or stops?

An inetd service cannot “depend” on a non-inetd service; a D-Bus service
cannot depend on a non-D-Bus service.  Both D-Bus and inetd have their
own notion of what a service is, how to start it, etc., which is
separate from what the Shepherd does.

I reckon that calling everything a “service” does not help understand
all this…

HTH,
Ludo’.

Re: Services and log management/monitoring

[rendaw]

On 4/19/19 9:09 PM, Ludovic Courtès wrote:
> Hi,
>
> rendaw <7e9wc56emjakcm@s.rendaw.me> skribis:
>
>> I think fundamentally what I'd most like to know is when should I use a
>> Shepherd service vs a non-Shepherd service.  Maybe it's as simple as: if
>> you don't have any specific requirements always define a Shepherd service.
> It’s hard to answer that question in the abstract.  Do you have an
> example in mind that we could work through?

Yeah, so the basic example I was thinking of is I have an executable
binary (a server for diagnostics) I'd like to run when the computer boots.

>
>> Beyond what a service actually is though I have a few more questions:
>>
>> * Both
>> https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html
>> and
>> https://www.gnu.org/software/guix/manual/en/html_node/Shepherd-Services.html#Shepherd-Services
>> appear to show a dependency graph.
> The first page shows a service extension graph.
>
> The second page shows a graph of dependencies among Shepherd services.
>
> These are two different beasts.

Ah okay, that explains a lot - I was completely misinterpreting the
extension graph then.  I'm afraid I don't know what extension means in
this context.  Is it similar to OOP extension, where a "Dog" is-a
"Canine"?  Looking at the graphic, "accounts" depends-on "etc" (the
directory/mount?) makes a lot more sense to me than "accounts" is-a
"etc".  Similarly I would assume that the udev service runs udevd, but
if upower and colord are both "udev" services (because they extend udev)
does that mean if I run both the upower and colord services then my
system will have 3 instances of udevd running?

I'm fairly sure that's not what it means, but if extends isn't an
inheritance relationship and it's not a dependency relationship I'm not
sure what it is.

Is the udev service in the shepherd graph different then the udev
service (-type?) in the service extension graph?  That might be a factor
in my confusion.

>>   Are the dependency graphs Shepherd and non-Shepherd services
>> entirely separate?  Or maybe I'm completely misunderstanding
>> "extension" in this context.  Can an inet service depend on a non-inet
>> service?  Can an inet service depend on a d-bus service?  * Is there a
>> way to hook into service events - that is, run some code when a
>> service starts or stops?
> An inetd service cannot “depend” on a non-inetd service; a D-Bus service
> cannot depend on a non-D-Bus service.  Both D-Bus and inetd have their
> own notion of what a service is, how to start it, etc., which is
> separate from what the Shepherd does.

So suppose I have an mcron job that needs an ssh tunnel managed by
shepherd to be started, or an inetd handler that needs data files on a
network mount to run... is it possible to express those ordering
dependencies?  Or a service that subscribes the server to external
webhooks, and needs the webhook listeners (inetd handlers) to be started
before registering.

I assume the answer to my 2nd question from the previous email - if
there's a way to run code when a service's state changes - is that there
currently isn't such hooking mechanism.

>
> I reckon that calling everything a “service” does not help understand
> all this…

I don't think it's an unreasonable choice of words :) - there's just a
lot that isn't fitting in mentally for me.

And thank you for your patience with the answers, I think I'm drawing
much closer to understanding all this.  Once I've figured it out I'll
definitely write something up in case someone else has the same issues.

Re: Services and log management/monitoring

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3079 bytes --]

rendaw <7e9wc56emjakcm@s.rendaw.me> writes:

> I'm fairly sure that's not what it means, but if extends isn't an
> inheritance relationship and it's not a dependency relationship I'm not
> sure what it is.

In this context, when we say that a service A extends a service B, my
understanding is that we mean that A changes (customizes) something
about B.  It is definitely not inheritance.  And because A changes
something about B, I don't think "A depends on B" is the best way to
think about it.

For example, the guix-service-type extends the account-service-type.
The former provides the latter with a list of user and group accounts.
In this way, the guix-service-type tells the account-service-type what
users and groups the guix service needs in order to run.  When you add a
service of the guix-service-type to your operating system declaration's
list of services (it's already there by default), the
account-service-type will know what users and groups are required by the
guix-service-type, and it will create them automatically.  We say the
guix-service-type "extends" the account-service-type because the former
provides the latter with a list of users/groups to create.

That is what we mean by "extension".  Via this "extension" mechanism, it
is possible to make cross-cutting changes that affect many different
parts of the system, without making a bunch of changes to a bunch of
services.  For example, if the tor-service-type also needs to create a
user account and group in order to do its job, it can simply extend the
activation-service-type, just like the guix-service-type did.  And if
something about the way we create user accounts or groups needs to
change, we can change it for all users and groups just by modifying one
place: the account-service-type.

You might be wondering: if the account-service-type is extended by user
and group accounts, how are other services extended?  The answer is that
the type of object used depends on how the services are defined.  The
services participating in the extension define their own contract,
within the confines of the general "extension mechanism" that Guix
provides.  Many services document how they can be extended in the
manual, but some do not, and you might have to look at the source or
existing examples to figure it out, which is not so great.

Note that if you dig into the code, you will find that the term
"extension" can also refer to a specific record type called
<service-extension>, described in the "Service Reference" section of the
manual, and defined in the gnu/services.scm file.  If you want to learn
more about it, I suggest reading the manual and also the source code -
specifically the 'fold-services' procedure in gnu/services.scm, which is
where the actual extension mechanism is implemented.

The manual section is helpful, but due to some word ambiguities, it is a
little difficult to follow the first time through.  Some diagrams could
help a lot in explaining the concepts.  I hope this email helped a
little.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Services and log management/monitoring

[rendaw]

On 5/4/19 4:01 PM, Chris Marusich wrote:
> rendaw <7e9wc56emjakcm@s.rendaw.me> writes:
>
>> I'm fairly sure that's not what it means, but if extends isn't an
>> inheritance relationship and it's not a dependency relationship I'm not
>> sure what it is.
> In this context, when we say that a service A extends a service B, my
> understanding is that we mean that A changes (customizes) something
> about B.  It is definitely not inheritance.  And because A changes
> something about B, I don't think "A depends on B" is the best way to
> think about it.
>
> For example, the guix-service-type extends the account-service-type.
> The former provides the latter with a list of user and group accounts.
> In this way, the guix-service-type tells the account-service-type what
> users and groups the guix service needs in order to run.  When you add a
> service of the guix-service-type to your operating system declaration's
> list of services (it's already there by default), the
> account-service-type will know what users and groups are required by the
> guix-service-type, and it will create them automatically.  We say the
> guix-service-type "extends" the account-service-type because the former
> provides the latter with a list of users/groups to create.
>
> That is what we mean by "extension".  Via this "extension" mechanism, it
> is possible to make cross-cutting changes that affect many different
> parts of the system, without making a bunch of changes to a bunch of
> services.  For example, if the tor-service-type also needs to create a
> user account and group in order to do its job, it can simply extend the
> activation-service-type, just like the guix-service-type did.  And if
> something about the way we create user accounts or groups needs to
> change, we can change it for all users and groups just by modifying one
> place: the account-service-type.
>
> You might be wondering: if the account-service-type is extended by user
> and group accounts, how are other services extended?  The answer is that
> the type of object used depends on how the services are defined.  The
> services participating in the extension define their own contract,
> within the confines of the general "extension mechanism" that Guix
> provides.  Many services document how they can be extended in the
> manual, but some do not, and you might have to look at the source or
> existing examples to figure it out, which is not so great.
>
> Note that if you dig into the code, you will find that the term
> "extension" can also refer to a specific record type called
> <service-extension>, described in the "Service Reference" section of the
> manual, and defined in the gnu/services.scm file.  If you want to learn
> more about it, I suggest reading the manual and also the source code -
> specifically the 'fold-services' procedure in gnu/services.scm, which is
> where the actual extension mechanism is implemented.

Wow, thanks for taking the time to write this up!  I believe that's a
much clearer explanation - I actually figured it out by reading the
source code extensively, but I thought this thread had been abandoned. 
I've written up something similar here:
https://gitlab.com/rendaw/blog/blob/master/how_to_guix_for_those_who_dont.md

I think it would be great to have this information in the
documentation!  I'd also elaborate on what actually happens in the end
("If every service just extends other services, what does the root
service do?"), how stuff actually runs (since operating systems are
processes and not data after all -> I think the answer is that the boot
service, activation service, and shepherd service all run processes but
that leads to a lot of other obvious questions: what's the difference
between the boot and activation services - or what do each do and what
makes them suitable for those tasks, what order are processes started in
each of them, how does guix deal with existing files created by
activation processes when reconfiguring the system, etc), what is a
service vs a service type, and an explanation of the folding process and
how it relates to a service type definition.

> The manual section is helpful

I strongly believe the manual section is not helpful to anyone who
doesn't already know how Guix works.  The explanation of _everything_
related to services hinges on the word "extend" - and it's never
defined!  It's a novel definition in this context, it's not a matter of
ambiguity.  The definition of extends here is strongly tied to the
folding procedure so both of those need to be in the documentation for a
reader to gather an understanding of what's being discussed.

Reading the source code is a colossal hurdle for users, and especially
considering that this is a core service and the actual behavior is
influenced by and influences hundreds of other pieces of code (unlike
things like `(shell ...)` which get passed through to the end system
interacting with relatively few other processes).  Knowing the big
picture of how Guix works as a system, how components interact under the
hood, etc. is critical because it lets you make an informed decision
when you want to create a new service, it gives you hints on how to read
arcane build errors, when the resulting system doesn't work it lets you
know where things could be going wrong, and if you want to read more
about a particular subject you know what topics/search terms/articles
could be relevant.

I hope this doesn't come off too aggressive.  I view myself as a user at
this point and really want Guix to take off.  I spent the greater part
of three weeks trying how to figure out how to get my new (fairly
simple) Guix system running.  IMO if it takes most users (who want to do
more than install X and then run ./x) three full-time weeks, breathing
Guile, and source code diving to reach a usable point then Guix will
never be able to achieve critical mass.

Re: Services and log management/monitoring

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2841 bytes --]

rendaw <7e9wc56emjakcm@s.rendaw.me> writes:

> I've written up something similar here:
> https://gitlab.com/rendaw/blog/blob/master/how_to_guix_for_those_who_dont.md

Nice!  Thank you for sharing it.

> I think it would be great to have this information in the
> documentation!

I agree, but I am not sure how to merge the important parts with the
documentation in a way that does not clutter it up and make it even more
confusing than it already is.

> I'd also elaborate on what actually happens in the end
> ("If every service just extends other services, what does the root
> service do?"), how stuff actually runs (since operating systems are
> processes and not data after all -> I think the answer is that the boot
> service, activation service, and shepherd service all run processes but
> that leads to a lot of other obvious questions: what's the difference
> between the boot and activation services - or what do each do and what
> makes them suitable for those tasks, what order are processes started in
> each of them, how does guix deal with existing files created by
> activation processes when reconfiguring the system, etc), what is a
> service vs a service type, and an explanation of the folding process and
> how it relates to a service type definition.

Yes, explaining this could make for a good manual section or blog post.

>> The manual section is helpful
>
> I strongly believe the manual section is not helpful to anyone who
> doesn't already know how Guix works.

It's true that we could do a better job of introducing services in the
manual.

> Reading the source code is a colossal hurdle for users

I think this statement is perhaps over-broad, since "reading the source"
can mean "reading an operating system configuration file," which is
definitely easier than understanding Guix's services.  That said, I
agree that users shouldn't have to grok fold-services just to understand
how to use services in their operating system configuration.  The manual
tries to explain services, but it just doesn't get the job done, and we
should improve it.

> I hope this doesn't come off too aggressive.

I don't think it does, and I appreciate hearing your opinion.  I
actually felt exactly the same way as you, when I was first learning
about services.  It was frustrating.  As I learned, I submitted patches
to change the documentation where it was unclear or incorrect.  But it
still isn't good enough.  I've been thinking about trying to rewrite
that section in a way that makes it easier to understand services for a
first-time reader, but I haven't yet done it.

If you have ideas about how to improve the documentation, please
consider submitting a patch.  Coding can be hard, but so can
documentation.  Every little improvement can help!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Services and log management/monitoring

[rendaw]

On 5/4/19 5:24 PM, Chris Marusich wrote:
> I've been thinking about trying to rewrite
> that section in a way that makes it easier to understand services for a
> first-time reader, but I haven't yet done it.
>
Ah, I'm totally for that!  I thought though if my previous post wasn't
over the line, suggesting (or contributing) a complete rewrite would. 
I'd offer the section from my guide on services but as I mentioned
before I hand waved some stuff that I never managed to figure out
(activation/boot service task ordering, how they handle failures, etc).