(Exposing?) config files and non-start/stop operations

(Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Hello,

I'm writing a service for dirvish, and I realized that if I'm following
current guix service routes, I might not be able to run all the
operations I need to.  It seems that the current route for Guix is to
have your service write out a config that more or less becomes part of
the environment for starting / stopping a daemon via Shepherd.  But what
if that's not all you need to do?

Aside from just "running as a daemon", plenty of (especially
applications which manage state) will need to have other commands that
are unlikely to be run from shepherd.  For example:

 - Initializing a data store.  For example, in dirvish I need to run
   a command to initialize a "vault" where I will be storing my data.
 - Manually invoking a garbage collection utility.
 - Manually invoking an integrity check utility.
 - Possibly some side effect involving querying the network.
 - Running schema migrations.

All sorts of things!  Most of them (all?) involve state or side effects,
but plenty of our most important services will be "state overlords" of
some type.

So it seems to me that one of two things will be needed... either:

 - Expose the configuration file directly, possibly by putting in
   `${profile}/etc/foo'
 - Expose "wrapped" utilities.  For example, instead of invoking
   "dirvish" directly, I might invoke a wrapped dirvish.

What are our thoughts on how to deal with this?  This is going to be
very important when moving towards GuixSD as a platform for server
deployment.

 - Chris

PS: Note that for either of the above suggested methods, this will be
extra tricky when we have users running multiple services of the same
application.  For example, we might have users running multiple
Wordpress installs or mail daemons or MediaGoblin instances on the same
machine, and I'm not really sure how you'd find which thing you're
looking for there (but I guess you have the same problem with which
identifier would be used in shepherd start/stop anyway).

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Christopher Allan Webber writes:

> Hello,
>
> I'm writing a service for dirvish, and I realized that if I'm following
> current guix service routes, I might not be able to run all the
> operations I need to.  It seems that the current route for Guix is to
> have your service write out a config that more or less becomes part of
> the environment for starting / stopping a daemon via Shepherd.  But what
> if that's not all you need to do?
>
> Aside from just "running as a daemon", plenty of (especially
> applications which manage state) will need to have other commands that
> are unlikely to be run from shepherd.  For example:
>
>  - Initializing a data store.  For example, in dirvish I need to run
>    a command to initialize a "vault" where I will be storing my data.
>  - Manually invoking a garbage collection utility.
>  - Manually invoking an integrity check utility.
>  - Possibly some side effect involving querying the network.
>  - Running schema migrations.
>
> All sorts of things!  Most of them (all?) involve state or side effects,
> but plenty of our most important services will be "state overlords" of
> some type.
>
> So it seems to me that one of two things will be needed... either:
>
>  - Expose the configuration file directly, possibly by putting in
>    `${profile}/etc/foo'
>  - Expose "wrapped" utilities.  For example, instead of invoking
>    "dirvish" directly, I might invoke a wrapped dirvish.

Talking about this with Ricardo Wurmus on irc, the idea of launching
such a utility from shepherd itself came to mind.

So imagine you want to run a tool like this:

  foo-db --config=/path/to/foodb-config.cfg gc --aggressive

It looks like shepherd has an "action" slot/method:

  https://www.gnu.org/software/shepherd/manual/html_node/Slots-of-services.html#Slots-of-services
  https://www.gnu.org/software/shepherd/manual/html_node/Service-Convenience.html#Service-Convenience

So I wonder if we could add additional actions, and do something like
this:

  herd gc foo-db --aggressive

Or, even more lazy (but maybe not as good?):

  herd run-cmd foo-db gc --aggressive

Anyway, either of these examples would call the appropriate command but
implicitly pass in the --config parameter appropriately.

It looks like (gnu services shepherd) doesn't expose the "actions" slot,
but couldn't we do that?

This seems like the right route.  What do others think?

 - Chris

Re: (Exposing?) config files and non-start/stop operations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1898 bytes --]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Christopher Allan Webber writes:
>
>> Hello,
>>
>> I'm writing a service for dirvish, and I realized that if I'm following
>> current guix service routes, I might not be able to run all the
>> operations I need to.  

Why not?  I read the rest of your email but this wasn't clear to me.

>> It seems that the current route for Guix is to have your service
>> write out a config that more or less becomes part of the environment
>> for starting / stopping a daemon via Shepherd.  But what if that's
>> not all you need to do?
>>
>> Aside from just "running as a daemon", plenty of (especially
>> applications which manage state) will need to have other commands that
>> are unlikely to be run from shepherd.  For example:
>>
>>  - Initializing a data store.  For example, in dirvish I need to run
>>    a command to initialize a "vault" where I will be storing my data.
>>  - Manually invoking a garbage collection utility.
>>  - Manually invoking an integrity check utility.
>>  - Possibly some side effect involving querying the network.
>>  - Running schema migrations.
>>
>> All sorts of things!  Most of them (all?) involve state or side effects,
>> but plenty of our most important services will be "state overlords" of
>> some type.

Why do those activities need to be represented as actions in Shepherd?
If we're running a daemon or service that already exposes a mechanism
for manually running tasks like these, then can't we just use "the usual
mechanism" for doing it?  For example, if we're running a daemon that
already provides a script to perform garbage collection, can't we just
invoke that script?  It isn't clear to me why we would need to model
domain-specific actions like that in Shepherd, although I can see why it
might be convenient.  Am I missing something?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: (Exposing?) config files and non-start/stop operations

[Ludovic Courtès]

Hello!

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Christopher Allan Webber writes:
>
>> Hello,
>>
>> I'm writing a service for dirvish, and I realized that if I'm following
>> current guix service routes, I might not be able to run all the
>> operations I need to.  It seems that the current route for Guix is to
>> have your service write out a config that more or less becomes part of
>> the environment for starting / stopping a daemon via Shepherd.  But what
>> if that's not all you need to do?
>>
>> Aside from just "running as a daemon", plenty of (especially
>> applications which manage state) will need to have other commands that
>> are unlikely to be run from shepherd.  For example:
>>
>>  - Initializing a data store.  For example, in dirvish I need to run
>>    a command to initialize a "vault" where I will be storing my data.
>>  - Manually invoking a garbage collection utility.
>>  - Manually invoking an integrity check utility.
>>  - Possibly some side effect involving querying the network.
>>  - Running schema migrations.
>>
>> All sorts of things!  Most of them (all?) involve state or side effects,
>> but plenty of our most important services will be "state overlords" of
>> some type.
>>
>> So it seems to me that one of two things will be needed... either:
>>
>>  - Expose the configuration file directly, possibly by putting in
>>    `${profile}/etc/foo'
>>  - Expose "wrapped" utilities.  For example, instead of invoking
>>    "dirvish" directly, I might invoke a wrapped dirvish.
>
> Talking about this with Ricardo Wurmus on irc, the idea of launching
> such a utility from shepherd itself came to mind.
>
> So imagine you want to run a tool like this:
>
>   foo-db --config=/path/to/foodb-config.cfg gc --aggressive
>
> It looks like shepherd has an "action" slot/method:
>
>   https://www.gnu.org/software/shepherd/manual/html_node/Slots-of-services.html#Slots-of-services
>   https://www.gnu.org/software/shepherd/manual/html_node/Service-Convenience.html#Service-Convenience
>
> So I wonder if we could add additional actions, and do something like
> this:
>
>   herd gc foo-db --aggressive
>
> Or, even more lazy (but maybe not as good?):
>
>   herd run-cmd foo-db gc --aggressive
>
> Anyway, either of these examples would call the appropriate command but
> implicitly pass in the --config parameter appropriately.
>
> It looks like (gnu services shepherd) doesn't expose the "actions" slot,
> but couldn't we do that?
>
> This seems like the right route.  What do others think?

We could do that (and custom actions would probably be useful in other
contexts), but like the other Chris ;-), I’m not sure whether this is
necessary.

So it seems to be that the state-management commands (initialization
commands, DB schema migration commands, etc.) could be run either:

  1. By an “activation snippet”, by extending ‘activation-service-type’
     like many services do.

  2. By an auxiliary Shepherd service, say ‘dirvish-init’, that the main
     service, say ‘dirvish’, would depend on.

How does that sound?

HTH,
Ludo’.

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Chris Marusich writes:

>>>  - Initializing a data store.  For example, in dirvish I need to run
>>>    a command to initialize a "vault" where I will be storing my data.
>>>  - Manually invoking a garbage collection utility.
>>>  - Manually invoking an integrity check utility.
>>>  - Possibly some side effect involving querying the network.
>>>  - Running schema migrations.
>>>
>>> All sorts of things!  Most of them (all?) involve state or side effects,
>>> but plenty of our most important services will be "state overlords" of
>>> some type.
>
> Why do those activities need to be represented as actions in Shepherd?
> If we're running a daemon or service that already exposes a mechanism
> for manually running tasks like these, then can't we just use "the usual
> mechanism" for doing it?  For example, if we're running a daemon that
> already provides a script to perform garbage collection, can't we just
> invoke that script?  It isn't clear to me why we would need to model
> domain-specific actions like that in Shepherd, although I can see why it
> might be convenient.  Am I missing something?

So sure, we can run "foo-db gc" occasionally (though system
administrators sometimes have to run these kinds of commands by hand).
But what about "foo-db dumpdb"?  That's not something we just run on a
cronjob.  You need access to that command.  And in order for the command
to do the right thing, it might need access to the config file.

There are some other things where we can try to initialize or run
migrations automatically, but that stuff can be very dangerous to just
do implicitly.  Now note, I *do* think we want to handle some of this
stuff behind the scenes as much as possible so that users don't have to
think about it.  But have you ever done a *really big* database schema
migration?

We run into two challenges:
 - Now we're trying to "idempotently manage state", which it turns out
   is very hard (and the source of many bugs in devops tooling)
 - Some commands either need to be run manually occasionally, or are
   never automatically run (see the dumpdb example).

Does this make sense?
 - Chris

 

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Ludovic Courtès writes:

> Hello!
>
> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>> Christopher Allan Webber writes:
>>
>>> Hello,
>>>
>>> I'm writing a service for dirvish, and I realized that if I'm following
>>> current guix service routes, I might not be able to run all the
>>> operations I need to.  It seems that the current route for Guix is to
>>> have your service write out a config that more or less becomes part of
>>> the environment for starting / stopping a daemon via Shepherd.  But what
>>> if that's not all you need to do?
>>>
>>> Aside from just "running as a daemon", plenty of (especially
>>> applications which manage state) will need to have other commands that
>>> are unlikely to be run from shepherd.  For example:
>>>
>>>  - Initializing a data store.  For example, in dirvish I need to run
>>>    a command to initialize a "vault" where I will be storing my data.
>>>  - Manually invoking a garbage collection utility.
>>>  - Manually invoking an integrity check utility.
>>>  - Possibly some side effect involving querying the network.
>>>  - Running schema migrations.
>>>
>>> All sorts of things!  Most of them (all?) involve state or side effects,
>>> but plenty of our most important services will be "state overlords" of
>>> some type.
>>>
>>> So it seems to me that one of two things will be needed... either:
>>>
>>>  - Expose the configuration file directly, possibly by putting in
>>>    `${profile}/etc/foo'
>>>  - Expose "wrapped" utilities.  For example, instead of invoking
>>>    "dirvish" directly, I might invoke a wrapped dirvish.
>>
>> Talking about this with Ricardo Wurmus on irc, the idea of launching
>> such a utility from shepherd itself came to mind.
>>
>> So imagine you want to run a tool like this:
>>
>>   foo-db --config=/path/to/foodb-config.cfg gc --aggressive
>>
>> It looks like shepherd has an "action" slot/method:
>>
>>   https://www.gnu.org/software/shepherd/manual/html_node/Slots-of-services.html#Slots-of-services
>>   https://www.gnu.org/software/shepherd/manual/html_node/Service-Convenience.html#Service-Convenience
>>
>> So I wonder if we could add additional actions, and do something like
>> this:
>>
>>   herd gc foo-db --aggressive
>>
>> Or, even more lazy (but maybe not as good?):
>>
>>   herd run-cmd foo-db gc --aggressive
>>
>> Anyway, either of these examples would call the appropriate command but
>> implicitly pass in the --config parameter appropriately.
>>
>> It looks like (gnu services shepherd) doesn't expose the "actions" slot,
>> but couldn't we do that?
>>
>> This seems like the right route.  What do others think?
>
> We could do that (and custom actions would probably be useful in other
> contexts), but like the other Chris ;-), I’m not sure whether this is
> necessary.
>
> So it seems to be that the state-management commands (initialization
> commands, DB schema migration commands, etc.) could be run either:
>
>   1. By an “activation snippet”, by extending ‘activation-service-type’
>      like many services do.
>
>   2. By an auxiliary Shepherd service, say ‘dirvish-init’, that the main
>      service, say ‘dirvish’, would depend on.
> 
> How does that sound?

That's probably fine in most cases; if you read my reply to the other
Chris Marusich you'll see that there are still some cases where you
might need to run commands manually... eg a "dumpdb" type command.
Maybe that could be handled by #2, but it seeems like in a dumpdb type
command you might need to be able to pass in arguments.  It seems like
Shepherd actions are the right case for such a thing?

 - Chris

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Christopher Allan Webber writes:

> Chris Marusich writes:
>
>>>>  - Initializing a data store.  For example, in dirvish I need to run
>>>>    a command to initialize a "vault" where I will be storing my data.
>>>>  - Manually invoking a garbage collection utility.
>>>>  - Manually invoking an integrity check utility.
>>>>  - Possibly some side effect involving querying the network.
>>>>  - Running schema migrations.
>>>>
>>>> All sorts of things!  Most of them (all?) involve state or side effects,
>>>> but plenty of our most important services will be "state overlords" of
>>>> some type.
>>
>> Why do those activities need to be represented as actions in Shepherd?
>> If we're running a daemon or service that already exposes a mechanism
>> for manually running tasks like these, then can't we just use "the usual
>> mechanism" for doing it?  For example, if we're running a daemon that
>> already provides a script to perform garbage collection, can't we just
>> invoke that script?  It isn't clear to me why we would need to model
>> domain-specific actions like that in Shepherd, although I can see why it
>> might be convenient.  Am I missing something?
>
> So sure, we can run "foo-db gc" occasionally (though system
> administrators sometimes have to run these kinds of commands by hand).
> But what about "foo-db dumpdb"?  That's not something we just run on a
> cronjob.  You need access to that command.  And in order for the command
> to do the right thing, it might need access to the config file.
>
> There are some other things where we can try to initialize or run
> migrations automatically, but that stuff can be very dangerous to just
> do implicitly.  Now note, I *do* think we want to handle some of this
> stuff behind the scenes as much as possible so that users don't have to
> think about it.  But have you ever done a *really big* database schema
> migration?
>
> We run into two challenges:
>  - Now we're trying to "idempotently manage state", which it turns out
>    is very hard (and the source of many bugs in devops tooling)
>  - Some commands either need to be run manually occasionally, or are
>    never automatically run (see the dumpdb example).
>
> Does this make sense?
>  - Chris

FWIW, I do agree that in most cases, we can put these in the background
and automate them so that users don't need to see them.  Possibly that's
true for the key things I want to do for dirvish. :)  So we can and
should when we can!

Where we can't (a dropdb type command, or something like "borg mount")
I do think it would be nice to expose Shepherd actions.

 - Chris

Re: (Exposing?) config files and non-start/stop operations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

Hi Chris!

Thanks for entertaining my questions.

Christopher Allan Webber <cwebber@dustycloud.org> writes:

>> So sure, we can run "foo-db gc" occasionally (though system
>> administrators sometimes have to run these kinds of commands by hand).
>> But what about "foo-db dumpdb"?  That's not something we just run on a
>> cronjob.  You need access to that command.  And in order for the command
>> to do the right thing, it might need access to the config file.

I'm having trouble understanding the motivation here.  Can you clarify
why it's preferable to model the action as a Shepherd action, even
though a mechanism already exists to perform the action (e.g., "foo-db
dumpdb")?

I don't know a lot about Shepherd (yet!), but it seems like you COULD
model the action as a Shepherd action.  The question is: WHY would that
be better than just performing the action via "the usual" mechanism?  If
you can provide a more concrete example which illustrates the problem
for a specific service/daemon, it might help me to understand where
you're coming from.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Chris Marusich writes:

> Hi Chris!
>
> Thanks for entertaining my questions.
>
> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>>> So sure, we can run "foo-db gc" occasionally (though system
>>> administrators sometimes have to run these kinds of commands by hand).
>>> But what about "foo-db dumpdb"?  That's not something we just run on a
>>> cronjob.  You need access to that command.  And in order for the command
>>> to do the right thing, it might need access to the config file.
>
> I'm having trouble understanding the motivation here.  Can you clarify
> why it's preferable to model the action as a Shepherd action, even
> though a mechanism already exists to perform the action (e.g., "foo-db
> dumpdb")?
>
> I don't know a lot about Shepherd (yet!), but it seems like you COULD
> model the action as a Shepherd action.  The question is: WHY would that
> be better than just performing the action via "the usual" mechanism?  If
> you can provide a more concrete example which illustrates the problem
> for a specific service/daemon, it might help me to understand where
> you're coming from.

Sure, sorry I'm not being clear.

So say you wanted to run `foo-db dumpdb'.  Now while most foo-db setups
point to /var/lib/foodb/, you have your foo-db setup mounted on some
external or network storage or something, so it's mounted at
/mnt/dbdisk/foodb/.  With me so far?

And here's how we configured that, in our GuixSD configuration:

  (operating-system
    ;; ...
    (services (foo-db-service #:path "/mnt/dbdisk/foodb")))

Okay, cool!  So the way this works is that the foo-db-service procedure
sets up the service so that it passes in a configuration file that it
generated with the path set appropriately, as above.  So when shepherd
starts/stops this service, it uses this config, and it knows where the
data is.  (It probably does something like
  `foo-db run --config=/gnu/store/.../foodb.cfg')

So now you want to back up the database manually.  No problem!  You have
that `foo-db dumpdb' command.

But you run it... and it doesn't work!  foo-db assumed that your data
would be in the default location, /var/lib/foodb/.  The command you
*want* to run is something like:

  `foo-db dumpdb --config=/path/to/foodb.cfg'

Now... your GuixSD system has, somewhere buried in it, a config file
that it passes in when shepherd starts/stops the daemon.  It's somewhere
in /gnu/store/ ... but you don't know where it is!  GuixSD has cleverly
provided this config file to the closure of its own daemon managing
environment, but has given you no way to access it.

So when you try to back up your database, too bad!  It won't work!

Does that make sense now?

Anyway, I think exposing the `actions' slot in shepherd is the way to
go.  We could already use it for some things other than the theoretical
examples I've given.  Last night I wanted to do something like:

  sudo mcron --schedule=10

... which would, according to the mcron docs, show me the next ten
upcoming tasks that are scheduled!  I wanted to verify that I was
writing my mcron procedures right.  Likewise, my shepherd configuration
seems obscured (well sort of, I can figure it out by looking at
`ps auwwwx' but I don't want to do that).  It would be nice if I could
do:

  sudo herd list-schedule mcron 10

Anyway, hopefully I've made my case! :)
 
 - Chris

Re: (Exposing?) config files and non-start/stop operations

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Chris Marusich writes:
>
>>>>  - Initializing a data store.  For example, in dirvish I need to run
>>>>    a command to initialize a "vault" where I will be storing my data.
>>>>  - Manually invoking a garbage collection utility.
>>>>  - Manually invoking an integrity check utility.
>>>>  - Possibly some side effect involving querying the network.
>>>>  - Running schema migrations.
>>>>
>>>> All sorts of things!  Most of them (all?) involve state or side effects,
>>>> but plenty of our most important services will be "state overlords" of
>>>> some type.
>>
>> Why do those activities need to be represented as actions in Shepherd?
>> If we're running a daemon or service that already exposes a mechanism
>> for manually running tasks like these, then can't we just use "the usual
>> mechanism" for doing it?  For example, if we're running a daemon that
>> already provides a script to perform garbage collection, can't we just
>> invoke that script?  It isn't clear to me why we would need to model
>> domain-specific actions like that in Shepherd, although I can see why it
>> might be convenient.  Am I missing something?
>
> So sure, we can run "foo-db gc" occasionally (though system
> administrators sometimes have to run these kinds of commands by hand).
> But what about "foo-db dumpdb"?  That's not something we just run on a
> cronjob.  You need access to that command.  And in order for the command
> to do the right thing, it might need access to the config file.

Oh I see, so the main issue is getting a hand on that config file.

In that case, yes, a Shepherd action would a good way to achieve it.

It’s also a situation where adding the config file to /etc would be
reasonable (until Shepherd actions can actually be added :-)).

> There are some other things where we can try to initialize or run
> migrations automatically, but that stuff can be very dangerous to just
> do implicitly.  Now note, I *do* think we want to handle some of this
> stuff behind the scenes as much as possible so that users don't have to
> think about it.  But have you ever done a *really big* database schema
> migration?

I guess not ;-), but I think it may be best to simply prevent the DB
service from starting when a migration needs to happen, and let the user
handle it explicitly and restart the DB service when they’re done.

Ludo’.

Re: (Exposing?) config files and non-start/stop operations

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Anyway, I think exposing the `actions' slot in shepherd is the way to
> go.  We could already use it for some things other than the theoretical
> examples I've given.  Last night I wanted to do something like:
>
>   sudo mcron --schedule=10
>
> ... which would, according to the mcron docs, show me the next ten
> upcoming tasks that are scheduled!  I wanted to verify that I was
> writing my mcron procedures right.  Likewise, my shepherd configuration
> seems obscured (well sort of, I can figure it out by looking at
> `ps auwwwx' but I don't want to do that).  It would be nice if I could
> do:
>
>   sudo herd list-schedule mcron 10
>
> Anyway, hopefully I've made my case! :)

Definitely!  :-)  This one is a good example, too.

Ludo’.

Re: (Exposing?) config files and non-start/stop operations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

Hi Chris,

Thank you for taking the time to clarify the motivation!  It helped a
lot.

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> The command you *want* to run is something like:
>
>   `foo-db dumpdb --config=/path/to/foodb.cfg'
>
> Now... your GuixSD system has, somewhere buried in it, a config file
> that it passes in when shepherd starts/stops the daemon.  It's somewhere
> in /gnu/store/ ... but you don't know where it is!

This is the information I was missing.  If your service/daemon/tool
requires that information to be supplied on the command line, then I can
see why it would be convenient to model the task as a Shepherd action.
I think that would be useful.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Ludovic Courtès writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>> Chris Marusich writes:
>>
>>>>>  - Initializing a data store.  For example, in dirvish I need to run
>>>>>    a command to initialize a "vault" where I will be storing my data.
>>>>>  - Manually invoking a garbage collection utility.
>>>>>  - Manually invoking an integrity check utility.
>>>>>  - Possibly some side effect involving querying the network.
>>>>>  - Running schema migrations.
>>>>>
>>>>> All sorts of things!  Most of them (all?) involve state or side effects,
>>>>> but plenty of our most important services will be "state overlords" of
>>>>> some type.
>>>
>>> Why do those activities need to be represented as actions in Shepherd?
>>> If we're running a daemon or service that already exposes a mechanism
>>> for manually running tasks like these, then can't we just use "the usual
>>> mechanism" for doing it?  For example, if we're running a daemon that
>>> already provides a script to perform garbage collection, can't we just
>>> invoke that script?  It isn't clear to me why we would need to model
>>> domain-specific actions like that in Shepherd, although I can see why it
>>> might be convenient.  Am I missing something?
>>
>> So sure, we can run "foo-db gc" occasionally (though system
>> administrators sometimes have to run these kinds of commands by hand).
>> But what about "foo-db dumpdb"?  That's not something we just run on a
>> cronjob.  You need access to that command.  And in order for the command
>> to do the right thing, it might need access to the config file.
>
> Oh I see, so the main issue is getting a hand on that config file.
>
> In that case, yes, a Shepherd action would a good way to achieve it.
>
> It’s also a situation where adding the config file to /etc would be
> reasonable (until Shepherd actions can actually be added :-)).

True.  Though we still run, potentially, into problems where multiple
instances of some service are provided, eg multiple mediagoblin servers
or mail daemons or etc.  Though maybe a prefix/suffix could be provided
by the user to the service definition, so it would really shouw up like
/etc/mediagoblin-foosite.org

It might also be possible to do "herd config foo-db" with "config" as an
action and get back your config file to stdout.

Note: I'm interested still in exploring the shepherd actions stuff
still... though I did realize this morning that it wouldn't help in the
rare commands that have interactive input... there's no way to send
input/output in that way through the herd afaict!  Oh well, that's
probably pretty rare.

Speaking of I/O from commands, I wonder how you'd give any kind of
output back through an action to the herd?  Afaict the protocol supports
it and allows sending back "messages" that will be displayed, but
nothing uses it yet.  There's a <command-reply> record type that afaict
nothing uses at all.

>> There are some other things where we can try to initialize or run
>> migrations automatically, but that stuff can be very dangerous to just
>> do implicitly.  Now note, I *do* think we want to handle some of this
>> stuff behind the scenes as much as possible so that users don't have to
>> think about it.  But have you ever done a *really big* database schema
>> migration?
>
> I guess not ;-), but I think it may be best to simply prevent the DB
> service from starting when a migration needs to happen, and let the user
> handle it explicitly and restart the DB service when they’re done.
>
> Ludo’.

Yep!  Does require handing them a config, but I guess we've discussed
some reasonable ways to make that happen for now.

I did test out extending the etc-service-type this week; it was pretty
easy!

 - Chris

Re: (Exposing?) config files and non-start/stop operations

[Ludovic Courtès]

Howdy!

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:

[...]

>> It’s also a situation where adding the config file to /etc would be
>> reasonable (until Shepherd actions can actually be added :-)).
>
> True.  Though we still run, potentially, into problems where multiple
> instances of some service are provided, eg multiple mediagoblin servers
> or mail daemons or etc.

Sure, definitely.  All I meant was that populating /etc can be done as a
quick stop-gap measure when it makes sense, but it’s not a great
solution, notably because of the multiple-instance problem you describe.

> Note: I'm interested still in exploring the shepherd actions stuff
> still... though I did realize this morning that it wouldn't help in the
> rare commands that have interactive input... there's no way to send
> input/output in that way through the herd afaict!  Oh well, that's
> probably pretty rare.
>
> Speaking of I/O from commands, I wonder how you'd give any kind of
> output back through an action to the herd?  Afaict the protocol supports
> it and allows sending back "messages" that will be displayed, but
> nothing uses it yet.  There's a <command-reply> record type that afaict
> nothing uses at all.

<command-reply> is used for every reply sent by the daemon, in
(shepherd).

However, this hasn’t been thought to provide interactive commands and
such things; I’m not sure it would be a great idea to support
interactive commands, dunno.

The protocol currently is just: you connect, you send a request, you get
a reply, and you disconnect.  Actions are expected to be non-blocking.

Ludo’.

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Ludovic Courtès writes:

>> True.  Though we still run, potentially, into problems where multiple
>> instances of some service are provided, eg multiple mediagoblin servers
>> or mail daemons or etc.
>
> Sure, definitely.  All I meant was that populating /etc can be done as a
> quick stop-gap measure when it makes sense, but it’s not a great
> solution, notably because of the multiple-instance problem you describe.

Right.

>> Speaking of I/O from commands, I wonder how you'd give any kind of
>> output back through an action to the herd?  Afaict the protocol supports
>> it and allows sending back "messages" that will be displayed, but
>> nothing uses it yet.  There's a <command-reply> record type that afaict
>> nothing uses at all.
>
> <command-reply> is used for every reply sent by the daemon, in
> (shepherd).
>
> However, this hasn’t been thought to provide interactive commands and
> such things; I’m not sure it would be a great idea to support
> interactive commands, dunno.

Yeah, I didn't think a "feeding back input" option was likely to be
possible.  I understand why it isn't.

> The protocol currently is just: you connect, you send a request, you get
> a reply, and you disconnect.  Actions are expected to be non-blocking.

Okay, an expectation of non-blocking behaviour is useful to know.
Especially because I'm not sure it's a guarantee we can provide.  Eg,
imagine one of the previous commands, such as dumpdb or gc, on a really
large database.  That could block for a bit.

So shepherd actions are probably fine for something like "herd status
mcron" but for running slow and expensive operations, we need some way
to expose the config file.

SO, the two ways to do that seem to be:
 - Populate those configs in /etc/ (maybe providing prefix/suffix option
   for multiple instances)... probably ok since we expect situations
   that need these configs to be relatively rare.
 - We could also have a shepherd action like "herd config mediagoblin"
   that would merely spit out the configuration file path... so someone
   could do something like:
     $ foo-db dump-db --config `herd config foo-db`

WDYT?
 - Chris

Re: (Exposing?) config files and non-start/stop operations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2347 bytes --]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> So shepherd actions are probably fine for something like "herd status
> mcron" but for running slow and expensive operations, we need some way
> to expose the config file.

The speed and cost of an operation are orthogonal to whether or not that
operation requires access to a config file.

> SO, the two ways to do that seem to be:
>  - Populate those configs in /etc/ (maybe providing prefix/suffix option
>    for multiple instances)... probably ok since we expect situations
>    that need these configs to be relatively rare.

Putting stuff into /etc seems undesirable because (1) putting things
outside the immutable store seems like an invitation to meddle with the
files, and (2) the possibility of file name conflicts exists, which you
are already aware of.  However, this might be the simplest solution, so
if nothing else seems better, I think it would be reasonable to do.

>  - We could also have a shepherd action like "herd config mediagoblin"
>    that would merely spit out the configuration file path... so someone
>    could do something like:
>      $ foo-db dump-db --config `herd config foo-db`

This seems less desirable than putting things into /etc because it
doesn't seem to be in line with the intended use of Shepherd.  My
understanding is that Shepherd's job is to nanny the system's processes.
Responding to queries about the location of the services' config files
doesn't seem germane to that job.

As I see it, there is another possible solution: Modify the
service/daemon/tool so that it is no longer necessary to expose the
config file location in the first place.  I'm still not sure which
daemon/service we're talking about here, but surely the daemon/service
knows where its configuration file is when it starts up.  Surely it
could be made to remember that, if it doesn't already.  Surely a tool
could be made which queries the service/daemon for that information if
necessary.  Surely the service/daemon could be made to perform
operations like a database dump without being told where its own
configuration file is.

If it's easier to modify the service/daemon/tool so that we don't have
to explicitly provide a config file, maybe we should do that instead of
trying to modify Shepherd.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: (Exposing?) config files and non-start/stop operations

[Ricardo Wurmus]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

>  - We could also have a shepherd action like "herd config mediagoblin"
>    that would merely spit out the configuration file path... so someone
>    could do something like:
>      $ foo-db dump-db --config `herd config foo-db`

Here’s another similar idea:

    herd environment <service> -- some command

The environment for a service may be represented by a bunch of
environment variables.  “some command” then runs within this
environment and can access the variables.

So if “foo-db” can read its configuration file from FOO_DB_CONFIG the
environment definition would be something like

    #~(("FOO_DB_CONFIG" . (string-append #$foo-db "/etc/config")))

In this case a user could do

    herd environment foo-db -- foo-db dump-db

If it doesn’t respect FOO_DB_CONFIG one could do

    herd environment foo-db -- 'foo-db dump-db --config $FOO_DB_CONFIG'

This is all just a crummy bash translation of a reader monad.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
http://elephly.net

Re: (Exposing?) config files and non-start/stop operations

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>>  - We could also have a shepherd action like "herd config mediagoblin"
>>    that would merely spit out the configuration file path... so someone
>>    could do something like:
>>      $ foo-db dump-db --config `herd config foo-db`
>
> Here’s another similar idea:
>
>     herd environment <service> -- some command
>
> The environment for a service may be represented by a bunch of
> environment variables.  “some command” then runs within this
> environment and can access the variables.
>
> So if “foo-db” can read its configuration file from FOO_DB_CONFIG the
> environment definition would be something like
>
>     #~(("FOO_DB_CONFIG" . (string-append #$foo-db "/etc/config")))

Problem is that in many cases the config file is passed via a
command-line option to the daemon.

Ludo’.

Re: (Exposing?) config files and non-start/stop operations

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:

[...]

>> The protocol currently is just: you connect, you send a request, you get
>> a reply, and you disconnect.  Actions are expected to be non-blocking.
>
> Okay, an expectation of non-blocking behaviour is useful to know.
> Especially because I'm not sure it's a guarantee we can provide.  Eg,
> imagine one of the previous commands, such as dumpdb or gc, on a really
> large database.  That could block for a bit.
>
> So shepherd actions are probably fine for something like "herd status
> mcron" but for running slow and expensive operations,

Right.

> we need some way to expose the config file.

Could be.

> SO, the two ways to do that seem to be:
>  - Populate those configs in /etc/ (maybe providing prefix/suffix option
>    for multiple instances)... probably ok since we expect situations
>    that need these configs to be relatively rare.
>  - We could also have a shepherd action like "herd config mediagoblin"
>    that would merely spit out the configuration file path... so someone
>    could do something like:
>      $ foo-db dump-db --config `herd config foo-db`
>
> WDYT?

Adding a ‘config’ action where we see fit would certainly make sense,
yes.  I guess it’ll have to be decided on a case-by-case basis, and
perhaps we’ll see that this pattern makes sense for a whole class of
services.

Ludo’.

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Chris Marusich writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> So shepherd actions are probably fine for something like "herd status
>> mcron" but for running slow and expensive operations, we need some way
>> to expose the config file.
>
> The speed and cost of an operation are orthogonal to whether or not that
> operation requires access to a config file.

But they aren't orthogonal to whether or not it's a shepherd action, if
a shepherd action is expected to be nonblocking.

>> SO, the two ways to do that seem to be:
>>  - Populate those configs in /etc/ (maybe providing prefix/suffix option
>>    for multiple instances)... probably ok since we expect situations
>>    that need these configs to be relatively rare.
>
> Putting stuff into /etc seems undesirable because (1) putting things
> outside the immutable store seems like an invitation to meddle with the
> files, and (2) the possibility of file name conflicts exists, which you
> are already aware of.  However, this might be the simplest solution, so
> if nothing else seems better, I think it would be reasonable to do.

Those are definitely concerns (though hopefully people wouldn't be
modifying things in /etc since "guix system reconfigure" will splat over
any changes).

>>  - We could also have a shepherd action like "herd config mediagoblin"
>>    that would merely spit out the configuration file path... so someone
>>    could do something like:
>>      $ foo-db dump-db --config `herd config foo-db`
>
> This seems less desirable than putting things into /etc because it
> doesn't seem to be in line with the intended use of Shepherd.  My
> understanding is that Shepherd's job is to nanny the system's processes.
> Responding to queries about the location of the services' config files
> doesn't seem germane to that job.

I agree that it seems a bit strange, but no solution really seems great.
However, I don't think it's totally outside the reasonable realm of
shepherd.  Shepherd actions are to execute some operation on a process
that's running (or which could run).  Returning the contextual
information of what config file is being used can fit that paradigm.
But it does feel a bit like a shoehorn.

> As I see it, there is another possible solution: Modify the
> service/daemon/tool so that it is no longer necessary to expose the
> config file location in the first place.  I'm still not sure which
> daemon/service we're talking about here, but surely the daemon/service
> knows where its configuration file is when it starts up.  Surely it
> could be made to remember that, if it doesn't already.  Surely a tool
> could be made which queries the service/daemon for that information if
> necessary.  Surely the service/daemon could be made to perform
> operations like a database dump without being told where its own
> configuration file is.

And where would the configuration file be?  And who would define it,
when?  That's effectively what imperative distributions do, and the
application "knows" to look in /etc/, for some user-mutated file.

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Ludovic Courtès writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>>
>>>  - We could also have a shepherd action like "herd config mediagoblin"
>>>    that would merely spit out the configuration file path... so someone
>>>    could do something like:
>>>      $ foo-db dump-db --config `herd config foo-db`
>>
>> Here’s another similar idea:
>>
>>     herd environment <service> -- some command
>>
>> The environment for a service may be represented by a bunch of
>> environment variables.  “some command” then runs within this
>> environment and can access the variables.
>>
>> So if “foo-db” can read its configuration file from FOO_DB_CONFIG the
>> environment definition would be something like
>>
>>     #~(("FOO_DB_CONFIG" . (string-append #$foo-db "/etc/config")))
>
> Problem is that in many cases the config file is passed via a
> command-line option to the daemon.
>
> Ludo’.

Yeah, I'm not sure it would work.

But if we were looking for inspiration in what other things have been
done in Guix, we could do something along the lines of "guix system vm"
where it returns a path to a derivation with the appropriate wrapping
already done, something like:

  `herd cmd <service>` --some --args

It's a bit hacky in a different way though!

 - Chris

Re: (Exposing?) config files and non-start/stop operations

[Christopher Allan Webber]

Ludovic Courtès writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
> Adding a ‘config’ action where we see fit would certainly make sense,
> yes.  I guess it’ll have to be decided on a case-by-case basis, and
> perhaps we’ll see that this pattern makes sense for a whole class of
> services.
>
> Ludo’.

Ok!  Well, glad we had this discussion... if I bump into the right
itch-scratching moment maybe I'll get the exposing shepherd actions
implemented and we can go from there.  Otherwise others are welcome to!
:)

 - Chris

Re: (Exposing?) config files and non-start/stop operations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3520 bytes --]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Chris Marusich writes:
>
>> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>>
>>> So shepherd actions are probably fine for something like "herd status
>>> mcron" but for running slow and expensive operations, we need some way
>>> to expose the config file.
>>
>> The speed and cost of an operation are orthogonal to whether or not that
>> operation requires access to a config file.
>
> But they aren't orthogonal to whether or not it's a shepherd action, if
> a shepherd action is expected to be nonblocking.

I agree that's true.  I admittedly don't know a lot about Shepherd
(yet!), but I would think that as long as invoking "herd dump-db
my-database" doesn't block Shepherd from performing its duties, the
duration of that invocation shouldn't matter.

However, since Ludo' explicitly said that "actions are expected to be
non-blocking," I think we should probably keep the actions non-blocking.

>>> SO, the two ways to do that seem to be:
>>>  - Populate those configs in /etc/ (maybe providing prefix/suffix option
>>>    for multiple instances)... probably ok since we expect situations
>>>    that need these configs to be relatively rare.
>>
>> Putting stuff into /etc seems undesirable because (1) putting things
>> outside the immutable store seems like an invitation to meddle with the
>> files, and (2) the possibility of file name conflicts exists, which you
>> are already aware of.  However, this might be the simplest solution, so
>> if nothing else seems better, I think it would be reasonable to do.
>
> Those are definitely concerns (though hopefully people wouldn't be
> modifying things in /etc since "guix system reconfigure" will splat over
> any changes).

In my experience, people often meddle with the system config files
because it is often the path of least resistance for ad-hoc changes.  If
you put a scary warning in there like "YOUR CHANGES WILL BE OVERRIDDEN"
then maybe they're less likely to do it, but a mechanism to enforce that
(e.g., by storing the config file in the immutable store) is better.

>>>  - We could also have a shepherd action like "herd config mediagoblin"
>>>    that would merely spit out the configuration file path... so someone
>>>    could do something like:
>>>      $ foo-db dump-db --config `herd config foo-db`
>>
>> This seems less desirable than putting things into /etc because it
>> doesn't seem to be in line with the intended use of Shepherd.  My
>> understanding is that Shepherd's job is to nanny the system's processes.
>> Responding to queries about the location of the services' config files
>> doesn't seem germane to that job.
>
> I agree that it seems a bit strange, but no solution really seems great.
> However, I don't think it's totally outside the reasonable realm of
> shepherd.  Shepherd actions are to execute some operation on a process
> that's running (or which could run).  Returning the contextual
> information of what config file is being used can fit that paradigm.
> But it does feel a bit like a shoehorn.

After thinking about it a little more, I'm not so sure I agree with what
I originally said.  At this point, I'm inclined to agree with Ludo', who
said we should decide what to do for each service on a case-by-case
basis.  So, if adding an action makes more sense to you here, then let's
do that.  We can see if a pattern emerges over time for other services.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]