What is the philosophy behind shepherd?

What is the philosophy behind shepherd?

[Katherine Cox-Buday]

I must preface this email with the assurance that there is no agenda
behind my questions; only ignorance and curiosity. Please read it with
that in mind!

A couple weeks ago, I was watching a video called "The Tragedy of
Systemd"[1]. In it, Benno Rice discusses the need for a so-called
"system layer" which is responding to the many complicated signals
coming into a system from thing happening (e.g. networks becoming
available/unavailable, VPNs mucking with DNS and routing tables, etc.).
He characterizes systemd and things like it as something that lives
between kernel-space and user-space.

It really opened my eyes to why something like systemd exists rather
than sticking with the old-style init systems.

Does Shepherd take the stance that it is, or is to become a "system
layer"?

If so, one of the criticisms he has for systemd is that instead of
pulling in protocols for things (e.g. DNS), and allowing best-of-breed
software to handle the implementation, it has pulled in the
responsibility for implementation as well. Any thoughts on that?

[1] - https://www.youtube.com/watch?v=o_AIw9bGogo
-- 
Katherine

Re: What is the philosophy behind shepherd?

[znavko]

Hello, Katherine! I've tried a little systemd, openrc and now I am using GuixSD with shepherd.
Lot of people and developers use systemd.
Here it is a link you may get info from http://without-systemd.org/wiki/index.php/Main_Page
Here some theses https://ihatesystemd.com it was interesting for me cause I am not as deep in init systems.

I think need to define criterion to get appreciation of something. Think that layer is not a criteria.
Systemd works fine for some cases when you are sysadmin that want to control lot of things in one general interface - init system.

April 6, 2019 7:30 PM, "Katherine Cox-Buday" <cox.katherine.e@gmail.com> wrote:

> I must preface this email with the assurance that there is no agenda
> behind my questions; only ignorance and curiosity. Please read it with
> that in mind!
> 
> A couple weeks ago, I was watching a video called "The Tragedy of
> Systemd"[1]. In it, Benno Rice discusses the need for a so-called
> "system layer" which is responding to the many complicated signals
> coming into a system from thing happening (e.g. networks becoming
> available/unavailable, VPNs mucking with DNS and routing tables, etc.).
> He characterizes systemd and things like it as something that lives
> between kernel-space and user-space.
> 
> It really opened my eyes to why something like systemd exists rather
> than sticking with the old-style init systems.
> 
> Does Shepherd take the stance that it is, or is to become a "system
> layer"?
> 
> If so, one of the criticisms he has for systemd is that instead of
> pulling in protocols for things (e.g. DNS), and allowing best-of-breed
> software to handle the implementation, it has pulled in the
> responsibility for implementation as well. Any thoughts on that?
> 
> [1] - https://www.youtube.com/watch?v=o_AIw9bGogo
> --
> Katherine

Re: What is the philosophy behind shepherd?

[Katherine Cox-Buday]

znavko@disroot.org writes:

> Hello, Katherine! I've tried a little systemd, openrc and now I am
> using GuixSD with shepherd. Lot of people and developers use systemd.
> Here it is a link you may get info from
> http://without-systemd.org/wiki/index.php/Main_Page Here some theses
> https://ihatesystemd.com it was interesting for me cause I am not as
> deep in init systems.

While I would not characterize myself as a sysadmin, nor an init system
expert, I use systemd just about every day. I'm familiar with the
generalities of how it works, and why.

> I think need to define criterion to get appreciation of something.
> Think that layer is not a criteria. Systemd works fine for some cases
> when you are sysadmin that want to control lot of things in one
> general interface - init system.

I think it does more than to serve a unified interface to sysadmins. For
example, its built-in DNS server can dynamically update a `resolv.conf`
which some systems symlink `/etc/resolv.conf` to. This allows systemd to
handle signals of connecting to networks or VPNs, and update the systems
DNS servers dynamically. The acceptance of these types of signals and
the actions they spawn are what I think make up the "system layer" Benno
Rice was discussing.

-- 
Katherine

Re: What is the philosophy behind shepherd?

[znavko]

[-- Attachment #1: Type: text/plain, Size: 2942 bytes --]

Kathernie, you will get more conceptions carefully reading these definitions:
https://access.redhat.com/articles/754933 (https://access.redhat.com/articles/754933)
The systemd system and service manager is responsible for controlling how services are started,
stopped and otherwise managed on Red Hat Enterprise Linux 7 systems. By offering on-demand service
start-up and better transactional dependency controls, systemd dramatically reduces start up times.
As a systemd user, you can prioritize critical services over less important services.
... More than services: Instead of just managing services, systemd can manage several different unit types...

https://www.gnu.org/software/shepherd (https://www.gnu.org/software/shepherd)
The GNU Daemon Shepherd or GNU Shepherd, formerly known as GNU dmd, is a service manager that looks
after the herd of system services. It provides a replacement for the service-managing capabilities
of SysV-init (or any other init) with a both powerful and beautiful dependency-based system with a
convenient interface.

Of course, 'system manager' systemd pulls the responsibility of whatever you want even boot logs and more than services.
In Russian forum linux.org.ru people dislike it. But it is using in some FSF approved systems (f.e. trisquel).

Did you read LFS book? There, system is building from scratches. That way is when you have no those strange layers with 'system managers'.
April 6, 2019 9:21 PM, "Katherine Cox-Buday" <cox.katherine.e@gmail.com (mailto:cox.katherine.e@gmail.com)> wrote:
  znavko@disroot.org (mailto:znavko@disroot.org) writes:Hello, Katherine! I've tried a little systemd, openrc and now I am
using GuixSD with shepherd. Lot of people and developers use systemd.
Here it is a link you may get info from
http://without-systemd.org/wiki/index.php/Main_Page (http://without-systemd.org/wiki/index.php/Main_Page) Here some theses
https://ihatesystemd.com (https://ihatesystemd.com) it was interesting for me cause I am not as
deep in init systems. 

While I would not characterize myself as a sysadmin, nor an init system
expert, I use systemd just about every day. I'm familiar with the
generalities of how it works, and why.I think need to define criterion to get appreciation of something.
Think that layer is not a criteria. Systemd works fine for some cases
when you are sysadmin that want to control lot of things in one
general interface - init system. 

I think it does more than to serve a unified interface to sysadmins. For
example, its built-in DNS server can dynamically update a `resolv.conf`
which some systems symlink `/etc/resolv.conf` to. This allows systemd to
handle signals of connecting to networks or VPNs, and update the systems
DNS servers dynamically. The acceptance of these types of signals and
the actions they spawn are what I think make up the "system layer" Benno
Rice was discussing.

--
Katherine

[-- Attachment #2: Type: text/html, Size: 3685 bytes --]

Re: What is the philosophy behind shepherd?

[Christopher Lemmer Webber]

Katherine Cox-Buday writes:

> I must preface this email with the assurance that there is no agenda
> behind my questions; only ignorance and curiosity. Please read it with
> that in mind!
>
> A couple weeks ago, I was watching a video called "The Tragedy of
> Systemd"[1]. In it, Benno Rice discusses the need for a so-called
> "system layer" which is responding to the many complicated signals
> coming into a system from thing happening (e.g. networks becoming
> available/unavailable, VPNs mucking with DNS and routing tables, etc.).
> He characterizes systemd and things like it as something that lives
> between kernel-space and user-space.
>
> It really opened my eyes to why something like systemd exists rather
> than sticking with the old-style init systems.
>
> Does Shepherd take the stance that it is, or is to become a "system
> layer"?
>
> If so, one of the criticisms he has for systemd is that instead of
> pulling in protocols for things (e.g. DNS), and allowing best-of-breed
> software to handle the implementation, it has pulled in the
> responsibility for implementation as well. Any thoughts on that?
>
> [1] - https://www.youtube.com/watch?v=o_AIw9bGogo

I think one interesting thing, to me, is that I agree that this is
desirable, but I think a microkernel + services approach would be much
better architecture.  In such a world then shepherd's role would be to
bootstrap the key services needed to talk to each other.

In other words, I think systemd exists in many ways to make up for the
limitations of a monolithic kernel approach.  In that way, it makes
sense, but I think we could do better with a different fundamental
infrastructure.

But maybe I am wrong!

 - Chris

Re: What is the philosophy behind shepherd?

[Pjotr Prins]

On Sat, Apr 06, 2019 at 07:08:55PM -0400, Christopher Lemmer Webber wrote:
> In other words, I think systemd exists in many ways to make up for the
> limitations of a monolithic kernel approach.  In that way, it makes
> sense, but I think we could do better with a different fundamental
> infrastructure.

One way to view it is that systemd does a lot of 'magic'. I.e., there
are internal assumptions that are not that clear. A black box. 

For most users, or use cases, that is fine, but when it comes to
running robust and reliable systems you want to define a system with
its services as a reproducible 'expression', much in the spirit of
Guix.

That is where shepherd comes in.

Systemd reminds me too much of Microsoft Windows. Things just happen.

Pj.

Re: What is the philosophy behind shepherd?

[Adam Pribyl]

Shepherd is more like (sysv)init system, what's different is it is in 
guile, ie. you may easily extended or script it like you did with init via 
bash and like you can not with systemd because systemd only allows 
ComplicatedSetOfPredefinedCrypticWordsToDoAnything.

To my experience systemd is good as long as you do not hit anything 
authors did not considered important to take care of. Latest hit:
https://bugzilla.redhat.com/show_bug.cgi?id=1665931


On Sat, 6 Apr 2019, Katherine Cox-Buday wrote:

> I must preface this email with the assurance that there is no agenda
> behind my questions; only ignorance and curiosity. Please read it with
> that in mind!
>
> A couple weeks ago, I was watching a video called "The Tragedy of
> Systemd"[1]. In it, Benno Rice discusses the need for a so-called
> "system layer" which is responding to the many complicated signals
> coming into a system from thing happening (e.g. networks becoming
> available/unavailable, VPNs mucking with DNS and routing tables, etc.).
> He characterizes systemd and things like it as something that lives
> between kernel-space and user-space.
>
> It really opened my eyes to why something like systemd exists rather
> than sticking with the old-style init systems.
>
> Does Shepherd take the stance that it is, or is to become a "system
> layer"?
>
> If so, one of the criticisms he has for systemd is that instead of
> pulling in protocols for things (e.g. DNS), and allowing best-of-breed
> software to handle the implementation, it has pulled in the
> responsibility for implementation as well. Any thoughts on that?
>
> [1] - https://www.youtube.com/watch?v=o_AIw9bGogo
> -- 
> Katherine
>
>

Re: What is the philosophy behind shepherd?

[Ludovic Courtès]

Hello Katherine,

Katherine Cox-Buday <cox.katherine.e@gmail.com> skribis:

> A couple weeks ago, I was watching a video called "The Tragedy of
> Systemd"[1]. In it, Benno Rice discusses the need for a so-called
> "system layer" which is responding to the many complicated signals
> coming into a system from thing happening (e.g. networks becoming
> available/unavailable, VPNs mucking with DNS and routing tables, etc.).
> He characterizes systemd and things like it as something that lives
> between kernel-space and user-space.
>
> It really opened my eyes to why something like systemd exists rather
> than sticking with the old-style init systems.
>
> Does Shepherd take the stance that it is, or is to become a "system
> layer"?

The Shepherd in its current form is extremely simple, if not
simplistic.  ;-)  All it does is monitor processes.

It does, for instance, handle hardware or networking events.  Instead,
in Guix we just have shepherd spawn eudev, NetworkManager, dbus-daemon,
etc.  Each of these in turn takes care of these other kinds of events.

The downside is that shepherd has a very gross vision of what’s
happening on the system.  This is mitigated in Guix by the fact that the
system service graph does show the big picture.  The upside is that the
Shepherd is not responsible for any of these complex things.

Not sure if that qualifies as a philosophy; if it does, that might be
called “minimalism” or “separation of concerns.”  :-)

Ludo’.

Re: What is the philosophy behind shepherd?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 13046 bytes --]

Hi,

On Sat, 06 Apr 2019 14:30:07 -0500
Katherine Cox-Buday <cox.katherine.e@gmail.com> wrote:

> "system layer" which is responding to the many complicated signals
> coming into a system from thing happening (e.g. networks becoming
> available/unavailable, VPNs mucking with DNS and routing tables, etc.).
> He characterizes systemd and things like it as something that lives
> between kernel-space and user-space.
> 
> It really opened my eyes to why something like systemd exists rather
> than sticking with the old-style init systems.

There are some init system approaches:

* Ideally, there would be /etc/inittab, and only /etc/inittab .
Dependency resolution (if any) would be done in a shotgun approach (restart
services that break, so eventually the whole thing would work--it just does
a lot of service restarts in the beginning.  Furthermore,
it backs off on restarting so it didn't restart services too quickly.
It doesn't need pid files or anything like that--it just
remembered what was started and what terminated on its own :P).
That is clearly the MIT approach: Have something shitty but simple which does
the minimal thing.  That's it.  SysV init supports that but unfortunately
adds to it in the same process.  Why?
If you want to start a service supervisor, just invoke it in /etc/inittab - it's fine!

* SysV init: It has a directory (actually one per runlevel) with shell scripts
and those are started, in name order.  No dependency resolution and no idea of
what each shell script does.  No standardized library either.  No
parallelization.  No activation.  That's clearly bad.  I don't understand how
the SysV init afficionados can defend it--that's not a sane way to manage a
large system.
And if you want to manage a small system, use /etc/inittab .
For that matter, if you want to manage a large system, use /etc/inittab and
invoke a service manager in there.

* OpenRC: This init system has simple bash scripts BUT with included library
(in bash) for common things, including *very good* dependency resolution,
parallel startup, and activation.  That's all.  It just works.  Still one
big conflation of init and supervisor in one process.

* Shepherd: This init system is pretty similar to OpenRC.  There are a few
bootstrapping cases.  For example, in the beginning, the system Shepherd
cannot log to syslog because it didn't start syslog (yet?).

Other init systems do insane things to make this work, including writing their
own builtin syslog daemon or by starting syslog themselves (complicating the
management model to be understood by the user) etc.

Shepherd just logs to the kernel log until syslog comes up, IF it comes up
(in which case syslogd will copy the kernel log anyway--or not, shepherd
doesn't care).
That's clearly a simpler solution.

These kinds of cases keep coming up and it could get pretty bad--but I
think together with Guix, Shepherd is nice because the actual set of
shepherd services depends on your "operating-system" form, so the
complicated cases--like conflicting services--usually don't show up
at runtime because the runtime service set doesn't contain extra services
in the first place (started or not).

Also, you can connect to a REPL there at runtime and extend it/inspect it,
so debugging is nice (as opposed to all the others on the list where
debugging is non-existent).

* Systemd: They standardized the service definition which means the usual
ways of managing programs as service are now short three-liners and also
easily understood.  I like that part.  I don't like pretty much everything
else they did.  For example they had *horrible* security bugs like
"User=" setting being a "suggestion".  If it couldn't switch to that
user, it just started that service as root.  Problem?  What problem?
Similarly, a sane system would have a way to distinguish user ids from
user names since a user name "5353" is allowed in UNIX just fine.  They
don't distinguish.  What's more, *clearly* when you write "User=0foo"
you mean "root" /s.  It went on like this for quite some years and the
bugtracker showed that they don't understand security and apparently
can't use C correctly (using C correctly is very difficult) to the point
of Linus Torvalds stopping a main systemd developer from contributing to
the upstream kernel because they broke kernel interfaces.
Then the memory leaks, the complicated source code they have, the
extraneous DIY service reimplementations, the
dbus-interface-*all*-the-things (the usual IPC between kernel and user
space is text files in /sys, not object broker interfaces that
introduce another non-file namespace to manage).  I'm all for
standardizing, but I'd prefer if the people doing the standardizing
were actually knowledgeable in managing large systems securely.

(The first thing in securing something is to cut it the down to the
essentials.  Every source code line you can remove from the process,
do remove.  Systemd usually does the opposite--although lately they
at least have separate processes for their NIH services.  The latter
doesn't help when it's still monolithic in practise)

> Does Shepherd take the stance that it is, or is to become a "system
> layer"?

I don't think so.  It's not really set in stone, but why would it do that?

The whole idea is misguided.  There's nothing stopping a user service
from providing a character special device (see cuse, uio) and providing the
same interface the kernel would have.  Also, if the kernel people actually
looked at Plan 9 they'd have more stuff as plain read-write text files in
/sys instead of weird ioctl or netlink sockets (or instead of sockets in
general, I guess).  Those could be easily provided even by shell scripts
(by bind mounting something to /sys/foo/bar).

That's not what they propose, though.  Now they want the kernel to provide
dbus services, coupling everything with everything else, intermingling
complicated state.

> If so, one of the criticisms he has for systemd is that instead of
> pulling in protocols for things (e.g. DNS), and allowing best-of-breed
> software to handle the implementation, it has pulled in the
> responsibility for implementation as well. Any thoughts on that?

Yes, most of the time it's the not-invented-here syndrome in systemd.

Programs that are maintained for many years usually have very few bugs,
are slowly adapted to cover all reasonable use cases and the authors
get an intimate understanding of the problem domain and use that
understanding to make the design of the program match it well.

Replacing syslogd that has like >30 years of experience in it by your
own haphazard implementation seems... ill-advised.

systemd has integrated DNS, time server, hostname, readahead, /dev file
generator, quota support, backlight, timezone support, cgroup support,
cron, inetd, log rotation etcetcetc (stop it already).

Binary log files.  Why?  Anything wrong with compression?  Or filters?
What if they get corrupted (for example on an unexpected shutdown, one of
the *major* reasons for having log files in the first place)?

That said, large systems have other challenges and sometimes a more
complicated supervisor is necessary.

For example in order to have LTSP client support, your X server has to
do XDMCP.  For that to find your other hosts, your network has to be
set up and for Kerberos (login) to work your time synchronization has
to work.
In order to do time synchronization your network should be setup (in
order to use NTP).  In order to log correctly, syslog should requires
time synchronization, so network should be setup, so network setup
cannot log to syslog.  On the other hand, maybe you can have your
dhcp server give you the initial time, then you don't need the other
one so early (see
https://tools.ietf.org/id/draft-ogud-dhc-udp-time-option-00.html ).
The secure DNS will fail if the time is wrong so if NTP tries to
resolve the host by name it will fail if the time is wrong.
Sometimes the hostname comes from DHCP.  But that means syslog can't
use the hostname in local logfiles since otherwise syslog would require
the hostname to be set up first, which would require DHCP, which would
require networking, which would require syslog.

So clearly some kind of dependency resolution has to be available.
Also, trade-offs sometimes have to be made.  If syslog is
configured to log to another machine, you'd prioritize bringing the
network up to logging the network bringup.  If syslog is configured
to log to a local machine, maybe you don't care about accurate
timestamps either in which case you'd prioritize syslog above logging
the network bringup.  If you like accurate timestamps and local logfiles,
maybe have a simple initial-time getter that you can use without total
network configuration (like getting it from dhcp), then start up syslog
and then start the network and then the periodic time synchronizer.

As you can see what one needs is actual goals to be specifiable.  Nobody
I know does that or even allows to specify that.

Also, in a good design, the dhcp client service in shepherd would 
provide "dhcp-client" regardless of the actual client used so you can
replace it easily by another implementation.

Calling it "dhcpcd" or something would be ill-advised (having a comment
say so is OK, but just "provides '(dhcpcd)" is ill-advised).

Shepherd allows "provides '(dhcpcd dhcp-client)" for a shepherd service
which is nice.

Both "requires" and "provides" would list actual features the service
requires or provides, usually not just one thing.

For example, dnsmasq has both dns and dhcp server support (!), so its
"provides" entry would be "provides '(dns-server dhcp-server)" and
if someone would only require dns-server, shepherd would start up
dnsmasq anyway.

Likewise, "ntp-client" would provide "time" and "ntp-client", because that's
what it does.  It's an ntp client and in the end it will provide the time.

You could also provide time another way and it would be just as well.

So this part of the design of Shepherd is very nice, although our
dnsmasq service doesn't do the above for some reason.

In any case, service management is complicated nowadays and modularization
has downsides, among others dependency resolution.  Same here.

Then there is the problem of when to start up services.  If your
init system has every service known to man in its service list
(Guix doesn't), you probably don't want to start them all at once.
It's nice to have a facility to start services on demand.
With network services, this is possible using socket activation where
systemd will provide a dummy (inet or unix) socket, and, ONLY once
someone connects to it, actually start up the real handler.
Same principle for dbus service activation, but the dbus-daemon does it.

But you'd have the same effect if you had swap space and just let the
kernel swap out things which sit there doing nothing.  If you access the
socket, boom it gets swapped in again and continues.  Furthermore,
modern programs already lazy-load libraries and functions in those
libraries and lazy-allocate memory, so it's not wasting any time or
memory either.  What exactly is socket activation trying to solve?
If you want to have a load balancer, use xinetd or... a load balancer.

As for the video, "It's buggy"... "It's software".  Well, yeah, but why
not wait until the bugs are mostly gone before integrating it into all
the distributions?  What's the rush?  Also, I think it's a cultural
problem that they don't understand security.  So it's not just buggy,
but the bugs are *careless*.  If you make the failure modes nice a buggy
program won't crash your entire system or make it unusable.

I do use systemd.  At the university I work at (among other things)
we have a lot of systemd-using operating systems installed and many of
those act up in weird ways.  For example the user services touted in
the video hang your entire login--you can't get to the desktop--if one
of those user services hangs.  (That's a quite recent version).
I mean how did they even manage to do that?  That's what you get for
tightly coupling things.  Now the whole system is brittle.  Sheesh.
That's what you get for making systemd [both modular and] monolithic.

Sometimes dbus proxies from the user bus to the session bus, sometimes
it doesn't.  Seems to depend on the phase of the moon.  It differs
on the same machine without configuration changes, from restart to
restart.  And they want to bring that into the kernel.  Yeah, right.

See also http://skarnet.org/software/s6/systemd.html .

In any case, I think init should be as minimal as possible and not
include all those funny things in the first place.  System configuration
and init is *not* the same.  Why is it in the same executable, one
which will bring the system down if it crashes?  That makes no sense.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]