Re: Improving Shepherd

[Carlo Zancanaro <carlo@zancanaro.id.au>]

[-- Attachment #1: Type: text/plain, Size: 3208 bytes --]

A few people came to join me on Friday to think about Shepherd. 
Thanks Alex, Efraim, and Jelle.

We talked about a few different things that we'd like to achieve 
with Shepherd. The most significant and achievable things were, I 
think: user services, child process control, and 
concurrency/parallelism.

User services - Alex has already sent a patch to the list to allow 
generating user services from the Guix side. The idea is to 
generate a Shepherd config file, allowing a user to invoke 
shepherd manually to start their services. A further extension to 
this would be to have something like systemd's "user sessions", 
where the pid 1 Shepherd automatically starts a user's services 
when they log in.

Child process control - this is my personal frustration, where 
Shepherd loses track of processes that fork away (e.g. "emacs 
--daemon"). I barely know anything about Linux process management, 
but from my reading this can be solved through Linux namespaces 
(if user namespaces are available). Could someone who knows more 
about this let me know if that's a productive direction for me to 
investigate? Or tell me a better way to go about it?

Concurrency/parallelism - I think Jelle was planning to work on 
this, but I might be wrong about that. Maybe I volunteered? We're 
keen to see Shepherd starting services in parallel, where 
possible. This will require some changes to the way we start/stop 
services (because at the moment we just send a "start" signal to a 
single service to start it, which makes it hard to be parallel), 
and will require us to actually build some sort of real dependency 
resolution. Longer-term our goal should be to bring fibers into 
Shepherd, but Efraim mentioned that fibers doesn't compile on ARM 
at the moment, so we'll have to get that working first at least.

I mentioned an idea to the guys on Friday about how Shepherd 
should treat enabled/disabled services. I've thought about it some 
more, and I think it might work. The general idea is that Shepherd 
would always try to run an enabled service, and it would leave a 
disabled service as-is (unless it's needed to start another 
service). So it would kind of work like this:
 - if stopped and enabled: try to start service
 - if started and enabled: monitor, and restart service if it 
 fails
 - if retrying too often: disable this service, and all which 
 depend on it
 - else: only start if another enabled service depends on this one

This would mean that Shepherd could decide the best way to 
start/stop services, including doing so in parallel if possible.

So, there are our ideas! Any thoughts, or words of wisdom? 
Feedback is welcome.

Carlo

On Mon, Jan 29 2018, Carlo Zancanaro wrote:
> I'm keen to do some work on shepherd. Partially this is driven 
> by
> me using it to manage my user session and having it not always
> work right, and partially this is driven by me grepping the code
> for "FIXME" (which was slightly overwhelming). If anyone is keen
> to chat about it on Friday, please find me! I have some ideas
> about things I'd like to do, but I don't really have any idea 
> what
> I'm doing. Any help/advice/encouragement you can give me will be
> appreciated!
>
> Carlo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]