Dependencies between service extensions

Dependencies between service extensions

[conjaroy]

Greetings help-guix,

I've been a casual user of Nix for a couple of years and have decided to
test the waters with Guix. While I'm looking forward to spending time with
Lisp after many years away, my biggest impression is that Guix seems to
have well-documented interfaces in cases where Nix relies more on loose
conventions.

After reviewing the manual and some of the service definitions, I'd like a
better understanding of how to implement a common pattern. Let's say that I
have some application Foo that uses an external system for persistence,
like a SQL database. Before starting up service Foo I need to ensure both
that the database service is running and that the database instance for Foo
has been initialized, because Foo doesn't know how to initialize the
database on its own.

The first issue (how to ensure that the database service is up) seems to be
solved by adding a shepherd-root-service-type service extension that
declares a set of "requirements". And the second issue (performing
pre-startup initialization) seems to be handled by the
activation-service-type extension. So far so good.

But I couldn't find documentation on whether service activation scripts can
safely rely on other services that happen to be declared as requirements in
the shepherd-root-service-type extension. And while I found many activation
scripts that do simple things like modifying the filesystem, I couldn't see
any that interact directly with other services. However, I did see some
evidence of service extensions relying on the side effects of other service
extensions: a number of activation scripts call "getpwnam" for info on
system accounts that could exist only if the corresponding
account-service-type extension has already been executed.

So my questions are: could someone clarify best practices for initializing
state in Service A before Service B starts up? And is there anything about
the ordering/dependencies of a service's extensions that could be better
documented in the manual?

Thanks for all of your work on this project.

Jason

Re: Dependencies between service extensions

[conjaroy]

Does anyone have feedback on how to implement this type of dependency
correctly in a Guix service?

On Sun, Jun 7, 2020 at 11:43 AM conjaroy <conjaroy@gmail.com> wrote:

> Greetings help-guix,
>
> I've been a casual user of Nix for a couple of years and have decided to
> test the waters with Guix. While I'm looking forward to spending time with
> Lisp after many years away, my biggest impression is that Guix seems to
> have well-documented interfaces in cases where Nix relies more on loose
> conventions.
>
> After reviewing the manual and some of the service definitions, I'd like a
> better understanding of how to implement a common pattern. Let's say that I
> have some application Foo that uses an external system for persistence,
> like a SQL database. Before starting up service Foo I need to ensure both
> that the database service is running and that the database instance for Foo
> has been initialized, because Foo doesn't know how to initialize the
> database on its own.
>
> The first issue (how to ensure that the database service is up) seems to
> be solved by adding a shepherd-root-service-type service extension that
> declares a set of "requirements". And the second issue (performing
> pre-startup initialization) seems to be handled by the
> activation-service-type extension. So far so good.
>
> But I couldn't find documentation on whether service activation scripts
> can safely rely on other services that happen to be declared as
> requirements in the shepherd-root-service-type extension. And while I found
> many activation scripts that do simple things like modifying the
> filesystem, I couldn't see any that interact directly with other services.
> However, I did see some evidence of service extensions relying on the side
> effects of other service extensions: a number of activation scripts call
> "getpwnam" for info on system accounts that could exist only if the
> corresponding account-service-type extension has already been executed.
>
> So my questions are: could someone clarify best practices for initializing
> state in Service A before Service B starts up? And is there anything about
> the ordering/dependencies of a service's extensions that could be better
> documented in the manual?
>
> Thanks for all of your work on this project.
>
> Jason
>

Re: Dependencies between service extensions

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 5184 bytes --]

conjaroy <conjaroy@gmail.com> writes:

> Greetings help-guix,
>
> I've been a casual user of Nix for a couple of years and have decided to
> test the waters with Guix. While I'm looking forward to spending time with
> Lisp after many years away, my biggest impression is that Guix seems to
> have well-documented interfaces in cases where Nix relies more on loose
> conventions.
>
> After reviewing the manual and some of the service definitions, I'd like a
> better understanding of how to implement a common pattern. Let's say that I
> have some application Foo that uses an external system for persistence,
> like a SQL database. Before starting up service Foo I need to ensure both
> that the database service is running and that the database instance for Foo
> has been initialized, because Foo doesn't know how to initialize the
> database on its own.
>
> The first issue (how to ensure that the database service is up) seems to be
> solved by adding a shepherd-root-service-type service extension that
> declares a set of "requirements". And the second issue (performing
> pre-startup initialization) seems to be handled by the
> activation-service-type extension. So far so good.
>
> But I couldn't find documentation on whether service activation scripts can
> safely rely on other services that happen to be declared as requirements in
> the shepherd-root-service-type extension. And while I found many activation
> scripts that do simple things like modifying the filesystem, I couldn't see
> any that interact directly with other services. However, I did see some
> evidence of service extensions relying on the side effects of other service
> extensions: a number of activation scripts call "getpwnam" for info on
> system accounts that could exist only if the corresponding
> account-service-type extension has already been executed.
>
> So my questions are: could someone clarify best practices for initializing
> state in Service A before Service B starts up? And is there anything about
> the ordering/dependencies of a service's extensions that could be better
> documented in the manual?

To encode requirements for an activation script, I think you need to
declare a service type for it with appropriate requirements, and make
the start and stop actions "noop".  Then you can have other services
depend on the "activation service".

I did something similar in a service I'm working on that consists of
many different daemons.  To avoid having to run essentially the same
activation script on each, I created a "common" service that all daemons
depend upon.  It's fairly verbose (you don't need a record type), but
looks like this:

--8<---------------cut here---------------start------------->8---
;; This is a dummy service that all Ganeti daemons depend upon, mainly to
;; avoid having the same activation snippet on each.
(define-record-type* <ganeti-common-configuration>
  ganeti-common-configuration make-ganeti-common-configuration
  ganeti-common-configuration?
  (ganeti ganeti-common-configuration-ganeti            ;<package>
          (default ganeti))
  (directories ganeti-common-configuration-directories  ;list of strings
               (default '("/var/log/ganeti"
                          "/var/log/ganeti/kvm"
                          "/var/log/ganeti/os"
                          "/var/lib/ganeti/rapi"
                          "/var/lib/ganeti/queue"
                          "/var/run/ganeti/bdev-cache"
                          "/var/run/ganeti/socket"
                          "/var/run/ganeti/instance-disks"
                          "/var/run/ganeti/instance-reason"
                          "/var/run/ganeti/livelocks"))))

(define (ganeti-common-activation config)
  (let ((directories (ganeti-common-configuration-directories config)))
  #~(begin
      (use-modules (guix build utils))
      (for-each mkdir-p '#$directories))))

(define ganeti-common-service
  (lambda _
    (list (shepherd-service
           (documentation "Create the directories required by Ganeti.")
           (provision '(ganeti-common))
           (requirement '(file-systems))
           ;; Do nothing but the activation snippet, at least for now.
           (start #~(const #t))))))

(define ganeti-common-service-type
  (service-type (name 'ganeti-common)
                (extensions
                 (list (service-extension activation-service-type
                                          ganeti-common-activation)
                       ;; This service also installs Ganeti to the profile
                       ;; to make gnt-cluster, etc readily available.
                       (service-extension profile-service-type
                                          (compose list
                                                   ganeti-common-configuration-ganeti))
                       (service-extension shepherd-root-service-type
                                          ganeti-common-service)))
                (default-value (ganeti-common-configuration))
                (description
                 "This service creates directories used by other Ganeti
daemons.")))
--8<---------------cut here---------------end--------------->8---

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Dependencies between service extensions

[conjaroy]

Hello Marius,

Thanks a lot for your reply - I agree that the concept of a meta-service to
factor out common startup tasks seems logical.

Based on your example, it sounds like you're suggesting that if I have a
service that extends both activation-service-type and
shepherd-root-service-type, and if the Shepherd extension service defines
its own service dependencies (via shepherd-service-requirement), then the
activation-service-type extension script will run in a context where those
required services are active?

I've spent a few days investigating how the service graph works and here's
what I think is happening:

- Individually, activation-service-type and shepherd-root-service-type
concatenate their respective service extension scripts into a single script.

- These two service types in turn extend boot-service-type, so when
extensions of boot-service-type are processed, the combined
activation-service-type boot script is concatenated with the combined
shepherd-root-service-type boot script (in that order, apparently).

- Therefore, at startup, Guix executes scripts for all
activation-service-type extensions before starting any Shepherd services.

So while the idea of performing initialization as a separate service still
makes sense, I believe that any logic depending on other Shepherd services
needs to run as a shepherd-root-service-type extension instead of an
activation-service-type extension.

If any of the above is incorrect, I'd love to hear more!

Cheers,

Jason

On Tue, Jun 23, 2020 at 5:51 PM Marius Bakke <marius@gnu.org> wrote:

> conjaroy <conjaroy@gmail.com> writes:
>
> > Greetings help-guix,
> >
> > I've been a casual user of Nix for a couple of years and have decided to
> > test the waters with Guix. While I'm looking forward to spending time
> with
> > Lisp after many years away, my biggest impression is that Guix seems to
> > have well-documented interfaces in cases where Nix relies more on loose
> > conventions.
> >
> > After reviewing the manual and some of the service definitions, I'd like
> a
> > better understanding of how to implement a common pattern. Let's say
> that I
> > have some application Foo that uses an external system for persistence,
> > like a SQL database. Before starting up service Foo I need to ensure both
> > that the database service is running and that the database instance for
> Foo
> > has been initialized, because Foo doesn't know how to initialize the
> > database on its own.
> >
> > The first issue (how to ensure that the database service is up) seems to
> be
> > solved by adding a shepherd-root-service-type service extension that
> > declares a set of "requirements". And the second issue (performing
> > pre-startup initialization) seems to be handled by the
> > activation-service-type extension. So far so good.
> >
> > But I couldn't find documentation on whether service activation scripts
> can
> > safely rely on other services that happen to be declared as requirements
> in
> > the shepherd-root-service-type extension. And while I found many
> activation
> > scripts that do simple things like modifying the filesystem, I couldn't
> see
> > any that interact directly with other services. However, I did see some
> > evidence of service extensions relying on the side effects of other
> service
> > extensions: a number of activation scripts call "getpwnam" for info on
> > system accounts that could exist only if the corresponding
> > account-service-type extension has already been executed.
> >
> > So my questions are: could someone clarify best practices for
> initializing
> > state in Service A before Service B starts up? And is there anything
> about
> > the ordering/dependencies of a service's extensions that could be better
> > documented in the manual?
>
> To encode requirements for an activation script, I think you need to
> declare a service type for it with appropriate requirements, and make
> the start and stop actions "noop".  Then you can have other services
> depend on the "activation service".
>
> I did something similar in a service I'm working on that consists of
> many different daemons.  To avoid having to run essentially the same
> activation script on each, I created a "common" service that all daemons
> depend upon.  It's fairly verbose (you don't need a record type), but
> looks like this:
>
> --8<---------------cut here---------------start------------->8---
> ;; This is a dummy service that all Ganeti daemons depend upon, mainly to
> ;; avoid having the same activation snippet on each.
> (define-record-type* <ganeti-common-configuration>
>   ganeti-common-configuration make-ganeti-common-configuration
>   ganeti-common-configuration?
>   (ganeti ganeti-common-configuration-ganeti            ;<package>
>           (default ganeti))
>   (directories ganeti-common-configuration-directories  ;list of strings
>                (default '("/var/log/ganeti"
>                           "/var/log/ganeti/kvm"
>                           "/var/log/ganeti/os"
>                           "/var/lib/ganeti/rapi"
>                           "/var/lib/ganeti/queue"
>                           "/var/run/ganeti/bdev-cache"
>                           "/var/run/ganeti/socket"
>                           "/var/run/ganeti/instance-disks"
>                           "/var/run/ganeti/instance-reason"
>                           "/var/run/ganeti/livelocks"))))
>
> (define (ganeti-common-activation config)
>   (let ((directories (ganeti-common-configuration-directories config)))
>   #~(begin
>       (use-modules (guix build utils))
>       (for-each mkdir-p '#$directories))))
>
> (define ganeti-common-service
>   (lambda _
>     (list (shepherd-service
>            (documentation "Create the directories required by Ganeti.")
>            (provision '(ganeti-common))
>            (requirement '(file-systems))
>            ;; Do nothing but the activation snippet, at least for now.
>            (start #~(const #t))))))
>
> (define ganeti-common-service-type
>   (service-type (name 'ganeti-common)
>                 (extensions
>                  (list (service-extension activation-service-type
>                                           ganeti-common-activation)
>                        ;; This service also installs Ganeti to the profile
>                        ;; to make gnt-cluster, etc readily available.
>                        (service-extension profile-service-type
>                                           (compose list
>
>  ganeti-common-configuration-ganeti))
>                        (service-extension shepherd-root-service-type
>                                           ganeti-common-service)))
>                 (default-value (ganeti-common-configuration))
>                 (description
>                  "This service creates directories used by other Ganeti
> daemons.")))
> --8<---------------cut here---------------end--------------->8---
>