Guix shepherd user services

Guix shepherd user services

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 4752 bytes --]

I've been thinking about shepherd user services.

We just had a discussion on the topic:

	https://lists.gnu.org/archive/html/help-guix/2019-02/msg00010.html

and an interesting patch was sent a while ago:

	https://lists.gnu.org/archive/html/guix-devel/2018-02/msg00047.html.

Jelle first introduced me to this idea during the Guix days and it took
me a while to understand why it's pure awesomeness.  In short: just like
we can fully configure an operating-system in Guile and we don't have to
touch /etc manually, we could do the same with our user profile.  At
last, done with .bashrc / .profile to start user daemons, cron jobs,
whatever synchronization script you use...

It would also make it possible to fully _initialize_ a user profile (just like
we initialize the operating-system) with the desired files.  For instance, a
service could synchronize git repositories (e.g. your dotfiles), deploy it,
synchronize your emails, generate the mlocate cache, snapshot some btrfs drives,
you name it.

As a result, on a Guix install it would be possible to run

	guix service --install -f my-services.scm

let the magic happen, relog(?) et enjoy a 100% pre-configured user
profile.  How cool is that?

Benefits over traditional dotfiles or cron jobs:

- No more need for .xinitrc, .local/bin, etc.

- Declarative and uniformly programmable in Guile.

- Every service is uniformly controllable (start, restart, stop, status, etc.).
  It's not so convenient to do that with a daemon that was started from .xinitrc.

- Drags non-propagated inputs automatically, no need to polute the user profile
  (e.g. with redshift, xcape, etc.)

- Pre-configured services to choose from, as provided by Guix (unlike mcron).


My thoughts regarding the implementation design:

- Services can be enabled, disabled, started, restarted, stopped, listed,
  they can show details and documentation, they can run pre-defined actions.
  They are semantically very different from packages (in fact they often use
  packages, but not always).  I don't think it would be good to mix and match
  services and packages in a command like `guix package --list-installer`.  So I
  suggest we keep them separate.

- The user must be able to build and install services.  We could have the
  following behaviour then:

  `guix service --build redshift` would produce a
  /gnu/store/...-shepherd-redshit.scm

  `guix service --install redshift` would install it to ~/.guix-profile/services/redshift.scm.

  `guix service --remove redshift` would remove it from the above location.

  `guix service --upgrade=redshift` would rebuilt the service (and maybe
  redshift) and install the new version.

- How do we write custom services?  Well, just like we write custom packages and
  custom operating-system, in a channel or in GUIX_PACKAGE_PATH (which name
  becomes less and less meaningful).

- How do we update services (e.g. the redshift service when the redshift package
  just got updated)?  My understanding is that services are citizens similar to
  operating system declarations: they depend on the current Guix version and the
  user-defined definitions (e.g. channels).
  Then `guix pull && guix service --upgrade` will do.

- How do we start the user shepherd on login and load the service configuration?
  We need to be cautious to support foreign distributions.  On GuixSD, we could
  arrange /etc/profile to run

    shepherd --config ~/.guix-profile/service-manifest.scm

  on login.  On foreign distributions, users can add the above command to
  ~/.profile, or ~/.bashrc, etc.

- If the package manifest and the service manifest are separate, it follows that
  we now have 2 parameters to "reproducibly" define a user profile.  I think
  that's fine and it allows for composition (mix and match service manifests and
  package manifests).

- Currently I think it's not possible to just define shepherd services without
  enabling them.  Thus it's not possible to have a list of available services
  just like we have a list of packages.  While this might be OK for the system
  (but is it?), I think it would be nice if, as a user, we could do a

      guix service --list-available

  and pick services from there.  This is somewhat a separate issue but
  it would be nice to solve it.  I don't know enough about the shepherd
  internals to go into the details here.

- Should Guix drive the services with a `guix service` subcommand or should we
  leave it to `herd` run by the user?  Since we use `herd` for system services,
  I think it's more consistent to stick to `herd`.

I'd like to work on it with anyone interesting on the topic.  I know
some people I've worked on it before, so feel free to share your
progress :)

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix shepherd user services

[Gábor Boskovits]

Hello Pierre,

> It would also make it possible to fully _initialize_ a user profile (just like
> we initialize the operating-system) with the desired files.  For instance, a
> service could synchronize git repositories (e.g. your dotfiles), deploy it,
> synchronize your emails, generate the mlocate cache, snapshot some btrfs drives,
> you name it.

This would be awesome! How could we help you to make this happen?

Best regards,
g_bor

Re: Guix shepherd user services

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

And here is another discussion:

--8<---------------cut here---------------start------------->8---
Re: Invoking user shepherd; Was: Re: Defining *user* services in Guix:
https://lists.gnu.org/archive/html/guix-devel/2017-06/msg00132.html
--8<---------------cut here---------------end--------------->8---

> This would be awesome! How could we help you to make this happen?

First, brainstorm! :)  There are quite a few points that are still unanswered.

Second, gather the code we already have.  In my understanding, quite a few
people have worked on it already.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix shepherd user services

[Tanguy Le Carrour]

Hello Guix!

I've just read this thread [1] on the archive and I think it would be a
great feature! Any update on the topic?!

[1]: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00132.html

I would love to have some services started when I log in (tor,
transmission-daemon, mpd, gpg-agent… you name them!) without having to
rely on the root user and system configuration.

One other nice thing would be to have something like systemd timers [2]
to replace cron jobs. I use them to run things like mbsync and vdirsyncer.

[2]: https://wiki.archlinux.org/index.php/Systemd/Timers

I'm not sure that I'm able to write any (good) Guile code, but I can read
some and discuss things, if that helps!

Regards


-- 
Tanguy

Re: Guix shepherd user services

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

Tanguy,

Tanguy Le Carrour wrote:
> Hello Guix!
>
> I've just read this thread [1] on the archive and I think it 
> would be a
> great feature! Any update on the topic?!
>
> [1]: 
> https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00132.html
>
> I would love to have some services started when I log in (tor,
> transmission-daemon, mpd, gpg-agent… you name them!) without 
> having to
> rely on the root user and system configuration.

Note that this is already possible if you want it:

  ~ λ cat /home/nckx/.config/shepherd/init.scm 
  (load "services.scm")
  (register-services emacs gpg-agent ibus-daemon jackd)
  (action 'shepherd 'daemonize) ; send shepherd into background
  (for-each start (list emacs)) ; services to start automatically

  ~ λ cat /home/nckx/.config/shepherd/services.scm 
  (define emacs
    (make <service>
      #:provides '(emacs)
      #:requires '()
      #:start (make-system-constructor "emacs --daemon")
      #:stop (make-system-destructor "emacsclient --eval 
      \"(kill-emacs)\"")))
  ;; ...much more snipped...

  ~ λ grep shepherd /home/nckx/.xsession 
  shepherd	# user service manager


The discussion is about how to add it (and user stuff in general) 
to Guix, which is a big wriggly bag of tasty worms.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Guix shepherd user services

[Tanguy Le Carrour]

Hi Tobias

Thanks for your answers!


Le 04/29, Tobias Geerinckx-Rice a écrit :
> > I would love to have some services started when I log in […]
> Note that this is already possible if you want it:

Reading the Shepherd documentation, I was expecting that something like
this was possible, but, correct me if I'm wrong, it's not really documented,
is it?


>  ~ λ cat /home/nckx/.config/shepherd/init.scm  (load "services.scm")
>  (register-services emacs gpg-agent ibus-daemon jackd)
>  (action 'shepherd 'daemonize) ; send shepherd into background
>  (for-each start (list emacs)) ; services to start automatically
> 
>  ~ λ cat /home/nckx/.config/shepherd/services.scm  (define emacs
>    (make <service>
>      #:provides '(emacs)
>      #:requires '()
>      #:start (make-system-constructor "emacs --daemon")
>      #:stop (make-system-destructor "emacsclient --eval
> \"(kill-emacs)\"")))
>  ;; ...much more snipped...
> 
>  ~ λ grep shepherd /home/nckx/.xsession  shepherd	# user service manager

Cooooool! :-)
Did you write the "much more snipped" by yourself? If yes, is it
available somewhere? Else, is it documented somewhere? Did you
copy/paste it from another Guix file?

Stupid questions: why aren't you using service types? Whose responsibility
should it be to provide the service(-type) definitions (`service.scm`)?
The user?  Guix? Both? To me, it would make more sense to
have the service(-type) definitions come with the packages and
have them instantiated/configured by the user, like in the config file
used by `guix system reconfigure`? This would be in the file used
by `guix user`, wouldn't it?
If things were not clear to me, now they are all messed up in my head! ^_^'


> The discussion is about how to add it (and user stuff in general) to Guix,
> which is a big wriggly bag of tasty worms.

What do you mean by "how to add it"? Is it about having a `guix user`
command?
If I can achieve something without it, it would already be great.

Thanks again for your time!

Regards

-- 
Tanguy

Re: Guix shepherd user services

[Tanguy Le Carrour]

Hello!


Le 05/01, Tanguy Le Carrour a écrit :
> Le 04/29, Tobias Geerinckx-Rice a écrit :
> >  ~ λ cat /home/nckx/.config/shepherd/init.scm  (load "services.scm")
> >  (register-services emacs gpg-agent ibus-daemon jackd)
> >  (action 'shepherd 'daemonize) ; send shepherd into background
> >  (for-each start (list emacs)) ; services to start automatically
> > 
> >  ~ λ cat /home/nckx/.config/shepherd/services.scm  (define emacs
> >    (make <service>
> >      #:provides '(emacs)
> >      #:requires '()
> >      #:start (make-system-constructor "emacs --daemon")
> >      #:stop (make-system-destructor "emacsclient --eval
> > \"(kill-emacs)\"")))
> >  ;; ...much more snipped...
> > 
> >  ~ λ grep shepherd /home/nckx/.xsession  shepherd	# user service manager
> 
> Cooooool! :-)
> Did you write the "much more snipped" by yourself? If yes, is it
> available somewhere? Else, is it documented somewhere? Did you
> copy/paste it from another Guix file?

I've been playing with shepherd and service configurations and it's
actually easier than I thought it would be, at least for "simple"
services. Thanks for showing me the way! :-)

I love it so much that I started/stopped the emacs service (defined with
the snippet above) a few times. It works as expected, except for
the `defunct` processes that hang around. Each time I run `herd stop emacs`,
I get a new one, as if shepherd could not forget about the dead process.
This doesn't prevent me from re-starting the process and using the service,
but it looks… messy!

Am I doing something wrong?!

-- 
Tanguy

Re: Guix shepherd user services

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]

Tanguy,

My apologies for not replying sooner.  I fail at managing the 
scarce free time I have these days.

Tanguy Le Carrour wrote:
>> Cooooool! :-)
>> Did you write the "much more snipped" by yourself?

I snipped some definitions for other services, like ibus-daemon 
and gpg-agent, because I haven't used them in a long time (didn't 
want to risk confusing anyone with code that might've bitrotted), 
and they don't add any new concepts to the emacs example.  It's 
more of the same.  You're not missing anything.

>> If yes, is it available somewhere? Else, is it documented 
>> somewhere?
>> Did you copy/paste it from another Guix file?

Only in my home directory :-)  No on-line dotfiles like the other 
cool kids.

I figured this much out on my own, but my use of the Shepherd's 
probably pretty basic.

Its manual isn't as far along as Guix's and, to be honest, didn't 
make much sense to me until *after* learning what I know from 
reading the code & good old trial and error.

> I've been playing with shepherd and service configurations and 
> it's
> actually easier than I thought it would be, at least for 
> "simple"
> services. Thanks for showing me the way! :-)
>
> I love it so much that I started/stopped the emacs service 
> (defined with
> the snippet above) a few times.

Hours of fun!  Happy to have helped.

> It works as expected, except for the `defunct` processes that 
> hang
> around.  Each time I run `herd stop emacs`, I get a new one, as 
> if
> shepherd could not forget about the dead process.  This doesn't
> prevent me from re-starting the process and using the service, 
> but
> it looks… messy!
>
> Am I doing something wrong?!

No, you're right, I'd simply never checked this before.  Stopping 
emacs gives me… Zombie emacs.  And mu.

Kind regards,

T G-R


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Guix shepherd user services

[Tanguy Le Carrour]

Hi Tobias, hi Guix!


Le 05/04, Tobias Geerinckx-Rice a écrit :
> My apologies for not replying sooner.  I fail at managing the scarce free
> time I have these days.

I'm not much better, so no need to apologise! ;-)


> Tanguy Le Carrour wrote:
> No, you're right, I'd simply never checked this before.  Stopping emacs
> gives me… Zombie emacs.  And mu.

Just to make sure, I tried and checked with the "system wide" shepherd.
Everything seem to be OK, no Zombies lurking in the corner. But… I
noticed that `wpa-supplicant` was running and, as I don't need it on my
desktop, I stopped it:

    ~# herd stop wpa-supplicant
    Service ntpd has been stopped.
    Service avahi-daemon has been stopped.
    Service networking has been stopped.
    Service wpa-supplicant has been stopped.

?! I was not expecting `networking` to stop too.

    ~# herd status
    …
    Stopped:
     - avahi-daemon
     - networking
     - ntpd
     - term-auto
     - user-homes
     - wpa-supplicant

And event if `network` is listed as "stopped", I still have access
to the network!? This is not clear to me!? Does `networking` mean
something different?!

And if I start `network` again, `wpa-supplicant` comes up with it, but
not `avahi-daemon` and `ntpd`.

    # herd start networking
    Service wpa-supplicant has been started.
    Service networking has been started.

Not quite the behaviour I was expecting! But maybe I didn't understand
correctly how things were supposed to work! :-(

I'll dig a bit in the doc, but any input is welcomed!


-- 
Tanguy