Divvying up service definitions

Divvying up service definitions

[Bruno Victal]

Hi,

As the gnu/services and gnu/home/services grow, I think we should
consider divvying the services into stand-alone modules or
subdirectories.

Consider the ⌜dovecot-service-type⌝ in gnu/services/mail.scm: as of
commit 'd22d2a05c389207f8cdcf824be7738b1499a987c' this service
definition is nearly 1600 lines long, with the remainder of the file
comprising of four other services with rudimentary support.

It becomes troublesome working with such amalgamations as it makes it
hard to keep track of the used modules and bindings, especially when
define-configuration is used since the serializing procedures might be
used by various service definitions. Further complicating things is
'define-maybe', whose use monopolizes the predicate and serializers for
a particular service definition.

Now, I'm not saying that we should go and split everything into its own
module, I'm saying that we should be allowed to split some of them if
convenient (all subjective but I believe we can see that working with a
monolithic file in the kilolines where the interactions aren't obvious
is not fun, and that's without bringing in the hygienic issues
surrounding define-configuration and define-maybe).

Some considerations (using dovecot-service-type as an example):
* Splitting this as gnu/services/mail/dovecot.scm.
  We preserve the logical grouping of the services (with the addition
  that, for extremely comprehensive definitions, these can be neatly
  organized into subdirectories. (same structure seen with gnu/*.scm)
  A drawback is that 'use-service-modules' might not work with this
  although I wonder whether 'use-service-modules' & co. provide any
  value if we are already doing '(use-modules (gnu) …)' to begin with.
  They look redundant IMO.

* Splitting this as gnu/services/dovecot.scm.
  We keep it compatible with 'use-service-modules' at the cost of having
  a multitude of files under gnu/services, without any logical grouping
  (messy).

-- 
Thanks,
Bruno.

Re: Divvying up service definitions

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Bruno,

On Tue, Oct 24 2023, Bruno Victal wrote:

> Further complicating things is
> 'define-maybe', whose use monopolizes the predicate and serializers for
> a particular service definition.

I've dealt with that in the past and support your effort.

> * Splitting this as gnu/services/dovecot.scm.
>   We keep it compatible with 'use-service-modules' at the cost of having
>   a multitude of files under gnu/services, without any logical grouping

The number of services we offer strikes me as sufficiently small for
your "unsorted" scheme to remain easy to navigate.

Also, we already use this scheme for several services---such as rsync,
ssh, vnc, certbot, certbot, cgit, cups, ldap, lirc, sddm, avahi, mcron,
spice, auditd, sysctl, getmail, lightdm, and syncthing. I am not sure
it's worth a long discussion.

Moreover, categorizations are often ambigious and can make it harder to
locate a particular service definition.

While some services may remain narrowly bundled---as they are in nfs,
dbus, herd, hurd, samba, docker, ganeti, and shepherd---categorizations
often exist purely in the eye of the beholder. For example, does
Kerberos belong into its own category, as it does now, or is part of
'authentication', or perhaps 'security'?

In short, I would proceed and split the services if there is no further
comment on your request. It will make development easier.

For a transitional period, we could perhaps provide intermediate modules
in old places which re-export the service definitions that were moved,
but I'm not sure it's really necessary.

Thank you for your clean-up efforts!

Kind regards
Felix

Re: Divvying up service definitions

[Bruno Victal]

Hi Felix,

On 2023-10-24 18:54, Felix Lechner wrote:
> The number of services we offer strikes me as sufficiently small for
> your "unsorted" scheme to remain easy to navigate.

I can see your point here if we're to do estimates and interpolation
based on the growth of the services so far although I don't think it
would hurt to organize them.

> Also, we already use this scheme for several services---such as rsync,
> ssh, vnc, certbot, certbot, cgit, cups, ldap, lirc, sddm, avahi, mcron,
> spice, auditd, sysctl, getmail, lightdm, and syncthing. I am not sure
> it's worth a long discussion.

Agreed, the crux of the discussion is splitting service-definitions into
their own modules.
Sorting them can be decided later. (in a separate discussion if necessary)

> Moreover, categorizations are often ambigious and can make it harder to
> locate a particular service definition.
> 
> While some services may remain narrowly bundled---as they are in nfs,
> dbus, herd, hurd, samba, docker, ganeti, and shepherd---categorizations
> often exist purely in the eye of the beholder. For example, does
> Kerberos belong into its own category, as it does now, or is part of
> 'authentication', or perhaps 'security'?

I dont think there's any problem wrt categorization. For your Kerberos
example, either would be fine as they're not mutually exclusive.
(though I'd lean towards 'authentication' here)

We already see something similar with gnu/packages/… and it hasn't caused
much pain I believe. (the concerns there are mostly about cyclic modules
which I don't believe is relevant for services)
Again, the categorization doesn't have any impact on the code itself and
its purpose is to collect the modules into something more manageable so
even outright miscategorizations don't have any impact on the functionality
of the code.

> For a transitional period, we could perhaps provide intermediate modules
> in old places which re-export the service definitions that were moved,
> but I'm not sure it's really necessary.

Absolutely, there's mechanisms for re-export and we can already see their
use. (e.g. gnu/system/shadow.scm)

-- 
Furthermore, I consider that nonfree software must be eradicated.

Cheers,
Bruno.

Re: Divvying up service definitions

[Attila Lendvai]

> I dont think there's any problem wrt categorization. For your Kerberos
> example, either would be fine as they're not mutually exclusive.
> (though I'd lean towards 'authentication' here)

sure, but the crux of the issue here is how to improve code readability; i.e. would it make the humans working on the Guix codebase more effective.

if all the 3 options are equally reasonable, then some people will have to check two places before they find it in the third.

with such a codebase people will develop a habit of just using multi-file grep to find stuff... which is arguably a better strategy to begin with.

and it's worth pointing it out here that following grep'able patterns like `define.*kerberos` highly facilitates code navigation, and especially so for newcomers. this is one of the features of lisp(ers) that i badly miss in many other languages/cultures. and arguably, this is much more important than the placement of the definitions.

and while i'm ranting, another useful strategy is to give unique names to abstractions, and make sure that wherever these abstractions are employed, that name shows up in the source code; i.e. it can be grap'ped. IOW, names should not be e.g. programmatically assembled in DWIM'ish ways, unless it is a frequent enough pattern that the loss of grep'pability is worth it. or abstractions should not be hidden behind late binding, unless it's worth the additional loss of code readability.

ultimately, definitions shouldn't live in text files, but in a source code database, with proper search and projection tools in the editor (and the DVCS) that understand the graph nature of the source code. that would make this entire discussion moot, but we're not there yet.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“The meeting of two personalities is like the contact of two chemical substances: if there is any reaction, both are transformed.”
	— Carl Jung (1875–1961)

Re: Divvying up service definitions

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Bruno,

On Thu, Oct 26 2023, Bruno Victal wrote:

> Agreed, the crux of the discussion is splitting service-definitions into
> their own modules.

In the current naming scheme, the module (gnu services configuration) is
an outlier: It does not ship "configuration services" but provides
facilities that help build services.

Perhaps the module should move elsewhere.

Kind regards
Felix

Re: Divvying up service definitions

[Maxim Cournoyer]

Hi Bruno,

Bruno Victal <mirai@makinata.eu> writes:

> Hi,
>
> As the gnu/services and gnu/home/services grow, I think we should
> consider divvying the services into stand-alone modules or
> subdirectories.
>
> Consider the ⌜dovecot-service-type⌝ in gnu/services/mail.scm: as of
> commit 'd22d2a05c389207f8cdcf824be7738b1499a987c' this service
> definition is nearly 1600 lines long, with the remainder of the file
> comprising of four other services with rudimentary support.
>
> It becomes troublesome working with such amalgamations as it makes it
> hard to keep track of the used modules and bindings, especially when
> define-configuration is used since the serializing procedures might be
> used by various service definitions. Further complicating things is
> 'define-maybe', whose use monopolizes the predicate and serializers for
> a particular service definition.
>
> Now, I'm not saying that we should go and split everything into its own
> module, I'm saying that we should be allowed to split some of them if
> convenient (all subjective but I believe we can see that working with a
> monolithic file in the kilolines where the interactions aren't obvious
> is not fun, and that's without bringing in the hygienic issues
> surrounding define-configuration and define-maybe).
>
> Some considerations (using dovecot-service-type as an example):
> * Splitting this as gnu/services/mail/dovecot.scm.
>   We preserve the logical grouping of the services (with the addition
>   that, for extremely comprehensive definitions, these can be neatly
>   organized into subdirectories. (same structure seen with gnu/*.scm)
>   A drawback is that 'use-service-modules' might not work with this
>   although I wonder whether 'use-service-modules' & co. provide any
>   value if we are already doing '(use-modules (gnu) …)' to begin with.
>   They look redundant IMO.
>
> * Splitting this as gnu/services/dovecot.scm.
>   We keep it compatible with 'use-service-modules' at the cost of having
>   a multitude of files under gnu/services, without any logical grouping
>   (messy).

That's a great initiative!  I agree that multiple 'define-configuration'
services per file can be a bit messy, having to use prefixes everywhere,
making the definitions more verbose.

I don't have a strong preference of the caterogization of services, but
would perhaps prefer the first one (gnu/services/mail/dovecot.scm),
which could then make it easy to offer some interface as
gnu/services/mail.scm that'd re-export all that is needed (would that
work, or reintroduce the same top-level clashes?).

-- 
Thanks,
Maxim

Re: Divvying up service definitions

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3498 bytes --]

On Tue, Nov 07, 2023 at 10:56:11AM -0500, Maxim Cournoyer wrote:
> Hi Bruno,
> 
> Bruno Victal <mirai@makinata.eu> writes:
> 
> > Hi,
> >
> > As the gnu/services and gnu/home/services grow, I think we should
> > consider divvying the services into stand-alone modules or
> > subdirectories.
> >
> > Consider the ⌜dovecot-service-type⌝ in gnu/services/mail.scm: as of
> > commit 'd22d2a05c389207f8cdcf824be7738b1499a987c' this service
> > definition is nearly 1600 lines long, with the remainder of the file
> > comprising of four other services with rudimentary support.
> >
> > It becomes troublesome working with such amalgamations as it makes it
> > hard to keep track of the used modules and bindings, especially when
> > define-configuration is used since the serializing procedures might be
> > used by various service definitions. Further complicating things is
> > 'define-maybe', whose use monopolizes the predicate and serializers for
> > a particular service definition.
> >
> > Now, I'm not saying that we should go and split everything into its own
> > module, I'm saying that we should be allowed to split some of them if
> > convenient (all subjective but I believe we can see that working with a
> > monolithic file in the kilolines where the interactions aren't obvious
> > is not fun, and that's without bringing in the hygienic issues
> > surrounding define-configuration and define-maybe).
> >
> > Some considerations (using dovecot-service-type as an example):
> > * Splitting this as gnu/services/mail/dovecot.scm.
> >   We preserve the logical grouping of the services (with the addition
> >   that, for extremely comprehensive definitions, these can be neatly
> >   organized into subdirectories. (same structure seen with gnu/*.scm)
> >   A drawback is that 'use-service-modules' might not work with this
> >   although I wonder whether 'use-service-modules' & co. provide any
> >   value if we are already doing '(use-modules (gnu) …)' to begin with.
> >   They look redundant IMO.
> >
> > * Splitting this as gnu/services/dovecot.scm.
> >   We keep it compatible with 'use-service-modules' at the cost of having
> >   a multitude of files under gnu/services, without any logical grouping
> >   (messy).
> 
> That's a great initiative!  I agree that multiple 'define-configuration'
> services per file can be a bit messy, having to use prefixes everywhere,
> making the definitions more verbose.
> 
> I don't have a strong preference of the caterogization of services, but
> would perhaps prefer the first one (gnu/services/mail/dovecot.scm),
> which could then make it easy to offer some interface as
> gnu/services/mail.scm that'd re-export all that is needed (would that
> work, or reintroduce the same top-level clashes?).

I assume the define-maybe's aren't public, so I'd guess that shouldn't
cause a problem as long as they aren't exported.  There's some services,
like ntpd and openntpd, which reuse the service user/group between them,
I think with those being intentional about making sure there aren't
clashes, or making sure they line up, would also be a good choice.

Or moving the define-maybes to the top of the file and reusing them
between services. Is that a possibility?

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Divvying up service definitions

[Bruno Victal]

[-- Attachment #1.1: Type: text/plain, Size: 759 bytes --]

Hi Efraim,

On 2023-11-09 07:15, Efraim Flashner wrote:
> I assume the define-maybe's aren't public, so I'd guess that shouldn't
> cause a problem as long as they aren't exported.

They're not public but they override definitions within the same file
if more than one (define-maybe foo) is present (e.g. when (prefix bar-)
is used)

> Or moving the define-maybes to the top of the file and reusing them
> between services. Is that a possibility?

Due to their non-hygienic nature and the (prefix foo-) argument this
can't work. IMO we should look into replacing this define-maybe business
with something like SRFI-189 (by integrating it into Guile).

-- 
Furthermore, I consider that nonfree software must be eradicated.

Cheers,
Bruno.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Divvying up service definitions

[Maxim Cournoyer]

Hi Bruno,

Bruno Victal <mirai@makinata.eu> writes:

> Hi Efraim,
>
> On 2023-11-09 07:15, Efraim Flashner wrote:
>> I assume the define-maybe's aren't public, so I'd guess that shouldn't
>> cause a problem as long as they aren't exported.
>
> They're not public but they override definitions within the same file
> if more than one (define-maybe foo) is present (e.g. when (prefix bar-)
> is used)
>
>> Or moving the define-maybes to the top of the file and reusing them
>> between services. Is that a possibility?
>
> Due to their non-hygienic nature and the (prefix foo-) argument this
> can't work. IMO we should look into replacing this define-maybe business
> with something like SRFI-189 (by integrating it into Guile).

Haven't given much thought to the idea, but I've recently tried my hand
at adding new SRFIs to Guile, and developed some tooling such as a
'snarfi' script to aid with converting its HTML spec to Texinfo doc;
you'll find the result at:
<https://lists.gnu.org/archive/html/guile-devel/2023-12/msg00026.html>
and the script at
<https://lists.gnu.org/archive/html/guile-devel/2023-12/msg00019.html>.

I hope these are useful in your (or someone's) quest to add SRFI 189 or
others.

-- 
Thanks,
Maxim

Re: Divvying up service definitions

[Ludovic Courtès]

Hello!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>> * Splitting this as gnu/services/dovecot.scm.
>>   We keep it compatible with 'use-service-modules' at the cost of having
>>   a multitude of files under gnu/services, without any logical grouping
>>   (messy).
>
> That's a great initiative!  I agree that multiple 'define-configuration'
> services per file can be a bit messy, having to use prefixes everywhere,
> making the definitions more verbose.
>
> I don't have a strong preference of the caterogization of services, but
> would perhaps prefer the first one (gnu/services/mail/dovecot.scm),
> which could then make it easy to offer some interface as
> gnu/services/mail.scm that'd re-export all that is needed (would that
> work, or reintroduce the same top-level clashes?).

I’m all for “cleanups” as proposed, and I don’t have strong preferences
on how to do that.

I’d like us to make sure, though, that this is made in a
backward-compatible way: these module names and exported bindings are
part of the API that users refer to from the OS config file.  Renaming
things typically breaks user config, and updating it to use the new
names can be tedious if there are no messages explaining what to do.
‘define-deprecated’ helps, but perhaps we need something a little bit
more fancy now.

(It would be great if we could reach a level of backward-compatibility
close to what Emacs does: nice deprecation messages, recording when a
particular binding was introduced, and generally changing user-visible
things rather slowly.)

Ludo’.