unifying mcron and shepherd, service woes, improvements

unifying mcron and shepherd, service woes, improvements

[raingloom]

Just throwing this out there:
mcron is basically a very simple service manager, so, why not just move
all its timer functionality into shepherd (like systemd's timer units)
and then we can have a unified way to manage and debug services.

But also, shepherd debugging is kind of terrible too, at least on Guix,
so we (and if no one else gets on it that probably means I) should do
something about that.
For example, it's pretty nontrivial to figure out how to run a
service's command in isolation. So far I always had to find it in
/proc, which is not great. stracing a service that fails to start is
also rather tricky. The best I could come up with is to run strace in a
loop and pgrep for the PIDs I'm interested in. Same goes for GDB, which
was still bad in general last time I tried it.

Security could also be improved probably. Can we have an OCAP model?
I feel like I don't know enough to say, so this mail is meant more as a
conversation starter.

So, can we do better? What other issues are there with Shepherd? Are
they fundamental design problems or just plain old bugs?

Or is everyone else happy with the current design and it's just me who
can't use Shepherd properly? 😅

Re: unifying mcron and shepherd, service woes, improvements

[Ludovic Courtès]

Hi!

raingloom <raingloom@riseup.net> skribis:

> Just throwing this out there:
> mcron is basically a very simple service manager, so, why not just move
> all its timer functionality into shepherd (like systemd's timer units)
> and then we can have a unified way to manage and debug services.

Yes, we could—see below.

> But also, shepherd debugging is kind of terrible too, at least on Guix,
> so we (and if no one else gets on it that probably means I) should do
> something about that.
> For example, it's pretty nontrivial to figure out how to run a
> service's command in isolation. So far I always had to find it in
> /proc, which is not great. stracing a service that fails to start is
> also rather tricky. The best I could come up with is to run strace in a
> loop and pgrep for the PIDs I'm interested in. Same goes for GDB, which
> was still bad in general last time I tried it.

I agree it’s kind of ugly.  We could provide “actions” to display the
service’s command line, for instance, as in:

  herd command openssh

The limitation right now is that the Shepherd’s API is procedural:
there’s ‘make-forkexec-constructor’, which returns a procedure that runs
a command.  That “hides” the command from the <service> object.  That’s
not hard to fix, though we’ll have to think hard about compatibility.

> Security could also be improved probably. Can we have an OCAP model?
> I feel like I don't know enough to say, so this mail is meant more as a
> conversation starter.

I think “having an ocap model” sounds nice but also vague.  :-)

A concrete step we can take is follow the principle of least authority
(POLA) for more services.  That was the spirit of
<https://guix.gnu.org/en/blog/2017/running-system-services-in-containers/>.

> So, can we do better? What other issues are there with Shepherd? Are
> they fundamental design problems or just plain old bugs?
>
> Or is everyone else happy with the current design and it's just me who
> can't use Shepherd properly? 😅

I think it’s fair to say it’s rough on the edges.  :-)

One thing that’s on the to-do list is switching to a real event loop in
lieu of the current ad-hoc blocking design (this was discussed recently
on this mailing list).  The switch to ‘signalfd’ in the last release in
a step in that direction.  This will unlock “socket activation” and
possibly timers as you mentioned.

Thanks,
Ludo’.

Re: unifying mcron and shepherd, service woes, improvements

[raingloom]

On Sat, 15 May 2021 18:59:52 +0200
Ludovic Courtès <ludo@gnu.org> wrote:

> > Security could also be improved probably. Can we have an OCAP model?
> > I feel like I don't know enough to say, so this mail is meant more
> > as a conversation starter.  
> 
> I think “having an ocap model” sounds nice but also vague.  :-)
> 
> A concrete step we can take is follow the principle of least authority
> (POLA) for more services.  That was the spirit of
> <https://guix.gnu.org/en/blog/2017/running-system-services-in-containers/>.

By OCAP I was thinking of things like not having global state, or at
least not having it implicitly. Something more functional. The lambda
calculus supports the OCAP model, and Guix is all about being
functional, so service management should not be any different.

Would also be nice if replacing running components was supported, a la
Erlang. Maybe steal the supervisor tree idea as well.

Re: unifying mcron and shepherd, service woes, improvements

[Philip McGrath]

On 5/15/21 12:59 PM, Ludovic Courtès wrote:
> raingloom <raingloom@riseup.net> skribis:
>> Or is everyone else happy with the current design and it's just me who
>> can't use Shepherd properly? 😅
> 
> I think it’s fair to say it’s rough on the edges.  :-)
> 
> One thing that’s on the to-do list is switching to a real event loop in
> lieu of the current ad-hoc blocking design (this was discussed recently
> on this mailing list).  The switch to ‘signalfd’ in the last release in
> a step in that direction.  This will unlock “socket activation” and
> possibly timers as you mentioned.

 From a newcomer's perspective---I haven't used shepherd at all yet---I 
was surprised by the discussion leading up to 
<https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00396.html>:

On 3/19/21 10:42 PM, raid5atemyhomework wrote:
 > Perhaps the point is not so much to be *multi-threaded* but to be 
*concurrent*

 From very occasionally reading Andy Wingo's blog, I understand that 
Guile has a Concurrent ML implementation: 
<https://wingolog.org/archives/2017/06/29/a-new-concurrent-ml>.

I haven't programmed with it in Guile at all, but, from my experience 
using Racket's Concurrent ML system, I think it is the right foundation 
for a supervisor daemon. CML provides a consistent, extensible way to 
express, "when something happens, do some action", where "something 
happens" might be:

   * a timer fires;
   * a service, or a set of them, come up;
   * a subprocess exits (successfully or not);
   * a file descriptor is ready for reading and/or writing;
   * a "green thread" (IIUC, a Guile "fiber") has an uncaught exception;
   * a signal is received from the OS kernel;
   * a user sends a command from the client program;

... etc, etc, etc. The CML implementation handles many low-level details 
that are easy to get wrong.

-Philip

Re: unifying mcron and shepherd, service woes, improvements

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

On Sat, May 15 2021, raingloom wrote:

> Just throwing this out there:
> mcron is basically a very simple service manager, so, why not just move
> all its timer functionality into shepherd (like systemd's timer units)
> and then we can have a unified way to manage and debug services.

Yes please!

Not really related to Shepherd or Mcron, but having something like
Syslogd/Journald at the user level to browse and rotate logs would be
nice.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]