Composing service definitions (and maybe fmt)

Composing service definitions (and maybe fmt)

[Christopher Allan Webber]

Hello all,

So I'm thinking about all the service definitions I'm going to need to
put together, and just how many different types of config files there
are out there that we're going to need to support, and etc.  If we
really do produce nice scheme'y config representations for pretty much
everything, that's going to be a lot of writers... and there's a big
pushback (rightly so) in this group against "stringly typed" formatting
systems, so we can't take the Ansible/Salt/etc style jinja2 text
formatting style stuff.

So what's the plan?  It doesn't seem like there's a coherent pattern for
doing things between the different services, and some config files are
going to be quite complicated to write out.

Previously fmt has been mentioned as a possible direction to do string
writing without (intermediate) stringly-typed-templateness:

  http://synthcode.com/scheme/fmt/

I've booted up fmt in guile, it seems it works.  Maybe I could package
for Guix.  It's nice that it uses combinators to do things.  Maybe this
wouldn't be worth it for all things, but maybe it would be worth it for
some things.  It may be massive overkill.  I don't know.

I am impressed by how expressive the writer for javascript is in
it... snippet:

  ;;;; fmt-js.scm -- javascript formatting utilities
  ;;
  ;; Copyright (c) 2011-2012 Alex Shinn.  All rights reserved.
  ;; BSD-style license: http://synthcode.com/license.txt

  ;; ...

  (define (js-function . x)
    (let* ((name (and (symbol? (car x)) (car x)))
           (params (if name (cadr x) (car x)))
           (body (if name (cddr x) (cdr x))))
      (c-block
       (cat "function " (dsp (or name ""))  "("
            (fmt-join dsp params ", ") ")")
       (fmt-let 'return? #t (c-in-stmt (apply c-begin body))))))
  
  (define (js-var . args)
    (apply c-var 'var args))
  
  (define (js=== . args)
    (apply c-op "===" args))
  
... which could even be nicer with pattern matching, etc.
Again, that may be overkill, I don't know!

(I've also thought that some sort of string reader similar to skribe's
[foo ,(bar)] string-quasiquoting may make things easier.  Might even be
complimentary...)

What do people think?  Anyway there's also the concern of fmt as a
dependency maybe being tough for some people to get installed.  I dunno.
I could package fmt for guix though.  It might be worth knowing if
people think it's worth the time!

Alternately, what other direction should we go?  I'm a bit nervous to
get into services writing right now just because I'm not looking forward
to manually string-concatenating and string-joining things...

Thoughts?
 - Chris

Re: Composing service definitions (and maybe fmt)

[Ricardo Wurmus]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> (I've also thought that some sort of string reader similar to skribe's
> [foo ,(bar)] string-quasiquoting may make things easier.  Might even be
> complimentary...)

I like this idea.  It seems unrealistic to me to have configuration file
writers for each of the many ideosyncatic configuration file formats.  I
also don’t enjoy having to concatenate strings to generate configuration
files and I think that it would be useful if we had string-quasiquoting.

But I think we need some example cases first.  My experiences is
coloured by concatenating strings for udev rules.  That’s not nice, but
I also would not want to have to learn a new Schemey way of expressing
these rules — especially when I just want to copy them with minimal
changes from documentation and don’t want to have to understand them
first.

~~ Ricardo

Re: Composing service definitions (and maybe fmt)

[Christopher Allan Webber]

Ricardo Wurmus writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> (I've also thought that some sort of string reader similar to skribe's
>> [foo ,(bar)] string-quasiquoting may make things easier.  Might even be
>> complimentary...)
>
> I like this idea.  It seems unrealistic to me to have configuration file
> writers for each of the many ideosyncatic configuration file formats.

You're probably right.  It would be a lot of work to write that many
writers, and a lot to ingest every time.

> I also don’t enjoy having to concatenate strings to generate
> configuration files and I think that it would be useful if we had
> string-quasiquoting.

Yes, I think so.  It brings a lot of the advantages of other string
templating system without having some massively lossy DSl.

> But I think we need some example cases first.  My experiences is
> coloured by concatenating strings for udev rules.  That’s not nice, but
> I also would not want to have to learn a new Schemey way of expressing
> these rules — especially when I just want to copy them with minimal
> changes from documentation and don’t want to have to understand them
> first.

I agree that learning a bunch of writer functions for expressing rules
would *usually* just be too much to learn.  So maybe fmt is total
overkill.  (You can build lesser string generator functions yourself
that can be combined together anyhow, even if they aren't quite as
combinator'y.)

Though I think skribe-style string quasiquoting is not hard for a
schemer to learn.  I picked it up almost immediately.  And it's a lot
cleaner than the jinja2 style string templating I've grown accustomed to
from python web/deployment land.

Re: Composing service definitions (and maybe fmt)

[Ricardo Wurmus]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

>> But I think we need some example cases first.  My experiences is
>> coloured by concatenating strings for udev rules.  That’s not nice, but
>> I also would not want to have to learn a new Schemey way of expressing
>> these rules — especially when I just want to copy them with minimal
>> changes from documentation and don’t want to have to understand them
>> first.
>
> I agree that learning a bunch of writer functions for expressing rules
> would *usually* just be too much to learn.  So maybe fmt is total
> overkill.  (You can build lesser string generator functions yourself
> that can be combined together anyhow, even if they aren't quite as
> combinator'y.)
>
> Though I think skribe-style string quasiquoting is not hard for a
> schemer to learn.  I picked it up almost immediately.  And it's a lot
> cleaner than the jinja2 style string templating I've grown accustomed to
> from python web/deployment land.

I think I didn’t express myself clearly: I’m in favour of skribe-style
string quasiquoting.  The “Schemey way of expressing” things like udev
rules that “I would not want to have to learn” was referring to using
some sort of DSL rather than quasiquoting.

But I’m still not clear on what exactly this means.  For udev rules, for
example, I probably would not have any need for “variable substitution”
at all; I’d just copy and paste them verbatim.

What kind of configuration files do need substitution?  What would that
look like?  Is it only to reference store items as in gexps?

~~ Ricardo

Re: Composing service definitions (and maybe fmt)

[Christopher Allan Webber]

Ricardo Wurmus writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> Though I think skribe-style string quasiquoting is not hard for a
>> schemer to learn.  I picked it up almost immediately.  And it's a lot
>> cleaner than the jinja2 style string templating I've grown accustomed to
>> from python web/deployment land.
>
> I think I didn’t express myself clearly: I’m in favour of skribe-style
> string quasiquoting.  The “Schemey way of expressing” things like udev
> rules that “I would not want to have to learn” was referring to using
> some sort of DSL rather than quasiquoting.

Oh okay.  Well I'm convinced that string-quasiquoting is the best away
to go for now anyway.

> But I’m still not clear on what exactly this means.  For udev rules, for
> example [...]

I'm not sure either so I'm just going to say: it's probably
overcomplexifying things.  String quasiquoting will simplify things
quite a bit enough.  Plus, if we need to worry about "escaping"
variables, it'll be easy enough to have scheme procedures which do the
appropriate escaping which can be integrated into the scheme quasiquote
templates.  I think that'll solve 95% (to make an arbitrary
guess-number) of our needs without getting too complex.

So the question is, how should skribe-style string quasiquoting be
introduced?  And is anyone interested in working on it, or should I do
it?

Ludo, you've actually done quite a bit of skribe-style-stuff work! :)
Do you think this is a good idea?  And what would be the right approach?
Depending on guile-reader may be a bit heavy for Guix (?), though maybe
writing a reader macro to do string quasiquoting will not be hard anyway.

 - Chris

Re: Composing service definitions (and maybe fmt)

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludo, you've actually done quite a bit of skribe-style-stuff work! :)
> Do you think this is a good idea?  And what would be the right approach?
> Depending on guile-reader may be a bit heavy for Guix (?), though maybe
> writing a reader macro to do string quasiquoting will not be hard anyway.

I think we don’t want to depend on guile-reader, and I’m rather against
reader macros, except on rare occasions like #' and #~, because they
could interfere with other extensions that people use.

The Skribe syntax is just literal-text-by-default, where

  [foo bar ,baz]

is equivalent to:

  (list "foo bar " baz)

It’s convenient when writing document, but I’m not sure it is so helpful
for config files.  For instance, ‘text-file*’ and ‘mixed-text-file’ get
close to that, but without the reader macro.  See xorg.scm for example.

To me, the question is more about choosing between writing configuration
file bindings and exposing upstream’s configuration file syntax, as was
discussed when Andy posted the Dovecot service.  (To which I don’t have
a better answer than: let’s see on a case-by-case basis.)

WDYT?

But yeah, maybe we need a concrete example to illustrate the various
options.  :-)

Thanks,
Ludo’.

Re: Composing service definitions (and maybe fmt)

[Thompson, David]

On Wed, Jan 20, 2016 at 5:13 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> To me, the question is more about choosing between writing configuration
> file bindings and exposing upstream’s configuration file syntax, as was
> discussed when Andy posted the Dovecot service.  (To which I don’t have
> a better answer than: let’s see on a case-by-case basis.)

How about this:

Procedures that return services should accept file-like objects when
configuration files are needed.  When it makes sense (like with
elogind, dovecot, etc.), we can provide special procedures that take a
Scheme data structure and "compile" that to a file-like object in the
service's native configuration language.  This way, we allow ourselves
to have the high-level Scheme configuration APIs we like while also
allowing the use of "raw" configuration files in situations where it
is unavoidable or the path of least resistance.

- Dave

Re: Composing service definitions (and maybe fmt)

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Wed, Jan 20, 2016 at 5:13 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> To me, the question is more about choosing between writing configuration
>> file bindings and exposing upstream’s configuration file syntax, as was
>> discussed when Andy posted the Dovecot service.  (To which I don’t have
>> a better answer than: let’s see on a case-by-case basis.)
>
> How about this:
>
> Procedures that return services should accept file-like objects when
> configuration files are needed.  When it makes sense (like with
> elogind, dovecot, etc.), we can provide special procedures that take a
> Scheme data structure and "compile" that to a file-like object in the
> service's native configuration language.  This way, we allow ourselves
> to have the high-level Scheme configuration APIs we like while also
> allowing the use of "raw" configuration files in situations where it
> is unavoidable or the path of least resistance.

I like it!  When we provide Scheme data structures, we necessarily have
a ‘foo-configuration->file’ procedure anyway, so it’s just a matter of
exposing it.

Ludo’.

Re: Composing service definitions (and maybe fmt)

[Christopher Allan Webber]

Ludovic Courtès writes:

> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> On Wed, Jan 20, 2016 at 5:13 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>>> To me, the question is more about choosing between writing configuration
>>> file bindings and exposing upstream’s configuration file syntax, as was
>>> discussed when Andy posted the Dovecot service.  (To which I don’t have
>>> a better answer than: let’s see on a case-by-case basis.)
>>
>> How about this:
>>
>> Procedures that return services should accept file-like objects when
>> configuration files are needed.  When it makes sense (like with
>> elogind, dovecot, etc.), we can provide special procedures that take a
>> Scheme data structure and "compile" that to a file-like object in the
>> service's native configuration language.  This way, we allow ourselves
>> to have the high-level Scheme configuration APIs we like while also
>> allowing the use of "raw" configuration files in situations where it
>> is unavoidable or the path of least resistance.
>
> I like it!  When we provide Scheme data structures, we necessarily have
> a ‘foo-configuration->file’ procedure anyway, so it’s just a matter of
> exposing it.
>
> Ludo’.

I like it, and have been thinking similar things... except... except!

You may remember my post to the list about how I'm interested in
service configurations which know about each others' variables.  I wrote
about why I think this is really important to solving deployment here:

  http://lists.gnu.org/archive/html/guix-devel/2015-09/msg00736.html

If we build procedures which go from procedure inputs -> text and that's
all that gets passed to the service, do we lose the ability to
introspect the arguments being used to set up the service?

Would it be possible to have services which themselves set up services?
Eg, imagine I had something imaginary like:

  (3dprinter-service
   #:model "lawlsbot-3000"
   #:fill-pattern "hex")

(Ignore whether or not this is a sane way to configure a 3d printer at
all at the moment, I'm just making something up)

So, this might actually build the "implied" service of:

  (3dprinter-raw-service
   "[printconfig]
model = lawlsbot-3000
fill-pattern = hex")

... this way, daring users can provide their own config file if our
config file builders aren't good enough, but it's still possible to
have some other service or tool read that the 3dprinter-service has a
model with lawlsbot-3000, so it can show it in a menu, or something.

Is this possible, or even sane? :)

 - Chris

Re: Composing service definitions (and maybe fmt)

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> You may remember my post to the list about how I'm interested in
> service configurations which know about each others' variables.  I wrote
> about why I think this is really important to solving deployment here:
>
>   http://lists.gnu.org/archive/html/guix-devel/2015-09/msg00736.html
>
> If we build procedures which go from procedure inputs -> text and that's
> all that gets passed to the service, do we lose the ability to
> introspect the arguments being used to set up the service?

Not necessarily: <service> object have a ‘parameters’ field to hold the
service’s arguments, and this can always be introspected and
manipulated.

It’s just that there’s a “preferred form” to manipulate these service
parameters.  For instance, a high-level struct is easy to deal with, an
argument list is OK but less convenient, and a file-like object is
pretty bad.

> Would it be possible to have services which themselves set up services?
> Eg, imagine I had something imaginary like:
>
>   (3dprinter-service
>    #:model "lawlsbot-3000"
>    #:fill-pattern "hex")
>
> (Ignore whether or not this is a sane way to configure a 3d printer at
> all at the moment, I'm just making something up)
>
> So, this might actually build the "implied" service of:
>
>   (3dprinter-raw-service
>    "[printconfig]
> model = lawlsbot-3000
> fill-pattern = hex")
>
> ... this way, daring users can provide their own config file if our
> config file builders aren't good enough, but it's still possible to
> have some other service or tool read that the 3dprinter-service has a
> model with lawlsbot-3000, so it can show it in a menu, or something.
>
> Is this possible, or even sane? :)

So you’re suggesting to systematically have a high-level interface as
well as a lower-level interface that gives access to the raw config
file, right?

The problem is that often, the service configuration does not to just
one config file.  Often it also translates into command-line options,
user accounts, etc.  For instance, ‘tor-service’ expects users to pass a
raw config file, but it also needs to create a user account whose name
is given in that config file, so it needs to control that part of the
config file.

That being said, it’s always possible to extend ‘etc-service-type’ and
give it raw config files.

Not offering any concrete solution but… does that make sense?  :-)

Thanks,
Ludo’.

Re: Composing service definitions (and maybe fmt)

[Christopher Allan Webber]

Ludovic Courtès writes:

> So you’re suggesting to systematically have a high-level interface as
> well as a lower-level interface that gives access to the raw config
> file, right?
>
> The problem is that often, the service configuration does not to just
> one config file.  Often it also translates into command-line options,
> user accounts, etc.  For instance, ‘tor-service’ expects users to pass a
> raw config file, but it also needs to create a user account whose name
> is given in that config file, so it needs to control that part of the
> config file.
>
> That being said, it’s always possible to extend ‘etc-service-type’ and
> give it raw config files.
>
> Not offering any concrete solution but… does that make sense?  :-)
>
> Thanks,
> Ludo’.

It makes sense... I get the feeling that in order for us to really get a
sense of what to do right here, we're going to have to forge ahead and
maybe even get it a bit wrong, which is ok!

It could even be that the thing I really want to build has to be build
with an abstraction layer another layer up.  Not the worst thing!