unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* A new paradigm for modifying operating system declarations
@ 2021-01-04 15:38 raid5atemyhomework
  2021-01-04 20:21 ` Taylan Kammer
  2021-01-04 23:26 ` Jan Wielkiewicz
  0 siblings, 2 replies; 7+ messages in thread
From: raid5atemyhomework @ 2021-01-04 15:38 UTC (permalink / raw)
  To: guix-devel@gnu.org

Hi guix-developers,

I'd like to propose an idea for constructing `<operating-system>` objects.

First, let me present the `decorate` form:

```scheme
(define-syntax decorate
  (syntax-rules ()
    ((decorate ((x ...)) a ...)
     (x ... a ...))
    ((decorate (x) a ...)
     (x a ...))
    ((decorate ((x ...) y ...) a ...)
     (x ... (decorate (y ...) a ...)))
    ((decorate (x y ...) a ...)
     (x (decorate (y ...) a ...)))))
```

Here's an example usage, instead of:

```scheme
(with-output-to-port (current-error-port)
  (lambda ()
    (system "echo" "an error occurred")))
```

We can use:

```scheme
(decorate ((with-output-to-port (current-error-port))
           (lambda ()))
  (system "echo" "an error occurred"))
```

The reason why this is relevant, is that when I was tying out https://issues.guix.gnu.org/45643 , I ended up having several changes to the base `operating-system` form:

```scheme
(define system-zfs (make-zfs-package linux-libre-5.4))
(operating-system
  ; ... other fields ...
  (kernel linux-libre-5.4)
  (kernel-loadable-modules (list (list system-zfs "module")))
  (packages (cons* system-zfs
                   ; ... other packages ...
                   %base-packages))
  (services (cons* (service zfs-loader-service-type system-zfs)
                   ; ... other services ...
                   %desktop-services)))
```

So, I imagined instead of exposing `make-zfs-package` and `zfs-loader-service-type` and requiring so many modifications to various fields of `operating-system` form, expose instead a `install-zfs` form, like so:

```scheme
(install-zfs
  (operating-system
    (kernel linux-libre-5.4)
    ; ... other fields ...
    ))
```

This `install-zfs` form would be defined as below:

```scheme
(define-public (install-zfs os)
  (define system-zfs (make-zfs-package (operating-system-kernel os)))
  (operating-system
    (inherit os)
    (location (operating-system-location os))
    (kernel-loadable-modules (cons (list system-zfs "module")
                                   (operating-system-kernel-loadable-modules os)))
    (packages (cons system-zfs
                    (operating-system-packages os)))
    (services (cons (sevice zfs-loader-service-type system-zfs)
                    (operating-system-services os)))))
```

This would install ZFS "correctly", by adding the module to kernel loadable modules, adding the package so that ZFS can be managed at runtime, and adding the service to load ZFS module and import ZFS pools.  The hope is that this reduces the scope for errors in defining the operating system.

On the other hand, if this kind of pattern becomes common, then consider:

```scheme
(install-foo
  (install-bar
    (install-zfs
       (operating-system
          #;...))))
```

Which brings us back to the `decorate` form, which reduces nestedness:

```scheme
(decorate (install-foo
           install-bar
           install-zfs
           operating-system)
   #;...)
```

This seems quite elegant to me.

Now for example we can consider that the `"zfs"` module supports various options as well, which would be put in a `/etc/modprobe.d/zfs.conf` file. We could consider for example that `install-zfs` could support an `options` keyed argument, which it will then add to the `modprobe` configuration file in an `etc-service-type` in a new service that it extends to the given `<operating-system>`.

```scheme
(define install-zfs-full
  (lambda* (#:key (options '()) os)
    #;...))
(define-public install-zfs
  (match-lambda
    ((os)
     (install-zfs-full #:os os))
    (rest
     (apply install-zfs-full rest))))
```

Then in a system configuration:

```scheme
(decorate ((install-zfs #:options '(("zfs_arc_max" 5000000000)) #:os)
           operating-system)
  #;...)
```

Something similar could be done for the `ddcci-driver-linux` example in the documentation for `kernel-module-loader-service-type`:

```scheme
(define-public install-ddcci
  (match-lambda
    ((os)
     (install-ddcci-full #:os os))
    (rest
     (apply install-ddcci-full rest))))
(define install-ddcci-full
  (lambda* (#:key (options '())) os)
    (define config-file
      (plain-file "ddcci.conf"
        (if (null? options)
            ""
            (string-join (cons "options ddcci" options)
                         " "))))
    (operating-system
      (inherit os)
      (location (operating-system-location os))
      (kernel-loadable-modules (cons ddcci-driver-linux
                                     (operating-system-kernel-loadable-modules os)))
      (services
        (cons* (service kernel-module-loader-service-type
                 '("ddcci" "ddcci_backlight"))
               (simple-service 'ddcci-config etc-service-type
                 `(("modprobe.d/ddcci.conf" ,config-file)))
               (operating-system-services os))))))
```

Then, in the system configuration file:

```scheme
(decorate (install-zfs
           (install-ddcci #:options '("dyndbg" "delay=120") #:os)
           operating-system)
   (name "example-system")
   #;...)
```

Obviously, it's called "decorate" since it's partly inspired by Python decorators, which use a similar-looking pattern, with different syntax.

What are your opinions?  Blech?  Yummy?  Is it worth exploring this paradigm for adding particularly complex features to an operating system definition?

Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: A new paradigm for modifying operating system declarations
  2021-01-04 15:38 A new paradigm for modifying operating system declarations raid5atemyhomework
@ 2021-01-04 20:21 ` Taylan Kammer
  2021-01-05  0:32   ` raid5atemyhomework
  2021-01-04 23:26 ` Jan Wielkiewicz
  1 sibling, 1 reply; 7+ messages in thread
From: Taylan Kammer @ 2021-01-04 20:21 UTC (permalink / raw)
  To: raid5atemyhomework, guix-devel@gnu.org

On 04.01.2021 16:38, raid5atemyhomework wrote:
> Hi guix-developers,
> 
> I'd like to propose an idea for constructing `<operating-system>` objects.
> 
> [... snip ...]
> 
> What are your opinions?  Blech?  Yummy?  Is it worth exploring this paradigm for adding particularly complex features to an operating system definition?

Hi,

Interesting idea.

First, let me point out a more conventional alternative to what your 
'decorate' macro does:

   (define (compose proc . rest)
     "Functional composition; e.g. ((compose x y) a) = (x (y a))."
     (if (null? rest)
         proc
         (let ((rest-proc (apply compose rest)))
           (lambda x
             (let-values ((x (apply rest-proc x)))
               (apply proc x))))))

This allows for something like:

   ((compose install-foo install-bar install-zfs)
    (operating-system ...))

Or perhaps cleaner:

   (define my-os-modifier (compose install-foo install-bar install-zfs))

   (my-os-modifier
    (operating-system ...))

If you need custom modifications within, you can do:

   (define my-os-modifier
     (compose install-foo
              (lambda (os) ...)
              install-bar))

It's more verbose, but doesn't "break" standard Scheme syntax as much. 
Function composition is conceptually pretty easy and probably more 
well-known than "decorators" (which I had never heard of, personally).

Fewer macros means the reader needs to keep fewer special rules in mind.

---

Secondly, I wonder if passing an OS declaration through various 
procedures that modify it is really the best approach in the first place.

For build phases, we have the 'modify-phases' syntax.  For services, 
there is 'modify-services'.  Maybe there should be a 'modify-os' kind of 
syntax.  (In other words, if we're going to invent new syntax, why not 
go all-out and create the most convenient syntax for the use-case.)

Just my thoughts.


- Taylan


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: A new paradigm for modifying operating system declarations
  2021-01-04 15:38 A new paradigm for modifying operating system declarations raid5atemyhomework
  2021-01-04 20:21 ` Taylan Kammer
@ 2021-01-04 23:26 ` Jan Wielkiewicz
  2021-01-05  0:46   ` raid5atemyhomework
  1 sibling, 1 reply; 7+ messages in thread
From: Jan Wielkiewicz @ 2021-01-04 23:26 UTC (permalink / raw)
  To: raid5atemyhomework; +Cc: guix-devel@gnu.org

Dnia 2021-01-04, o godz. 15:38:38
raid5atemyhomework <raid5atemyhomework@protonmail.com> napisał(a):

> Hi guix-developers,

Hello.

> ```scheme
> (install-zfs
>   (operating-system
>     (kernel linux-libre-5.4)
>     ; ... other fields ...
>     ))
> ```

I don't like this way of nesting the OS declaration inside of any other
expression.
> 
> ```scheme
> (install-foo
>   (install-bar
>     (install-zfs
>        (operating-system
>           #;...))))

This makes it even worse.

> Which brings us back to the `decorate` form, which reduces nestedness:
> 
> ```scheme
> (decorate (install-foo
>            install-bar
>            install-zfs
>            operating-system)
>    #;...)

Better but still don't like it.
Can't we put the os declaration into a variable and then pass it to a
procedure?
Say:

> ```scheme
> (define OS
>   (operating-system
>     (kernel linux-libre-5.4)
>     ; ... other fields ...
>     ))
>
> (install-zfs OS)
> (install-foo OS)
> (install-bar OS)
> (install-something OS)
>
> ```

> Thanks
> raid5atemyhomework
> 

Jan Wielkiewicz



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: A new paradigm for modifying operating system declarations
  2021-01-04 20:21 ` Taylan Kammer
@ 2021-01-05  0:32   ` raid5atemyhomework
  2021-01-05  6:12     ` Carlo Zancanaro
  0 siblings, 1 reply; 7+ messages in thread
From: raid5atemyhomework @ 2021-01-05  0:32 UTC (permalink / raw)
  To: Taylan Kammer; +Cc: guix-devel@gnu.org

Good morning Taylan,

> First, let me point out a more conventional alternative to what your
> 'decorate' macro does:
>
> (define (compose proc . rest)
> "Functional composition; e.g. ((compose x y) a) = (x (y a))."
> (if (null? rest)
> proc
> (let ((rest-proc (apply compose rest)))
> (lambda x
> (let-values ((x (apply rest-proc x)))
> (apply proc x))))))
>
> This allows for something like:
>
> ((compose install-foo install-bar install-zfs)
> (operating-system ...))
>
> Or perhaps cleaner:
>
> (define my-os-modifier (compose install-foo install-bar install-zfs))
>
> (my-os-modifier
> (operating-system ...))
>
> If you need custom modifications within, you can do:
>
> (define my-os-modifier
> (compose install-foo
> (lambda (os) ...)
> install-bar))
>
> It's more verbose, but doesn't "break" standard Scheme syntax as much.
> Function composition is conceptually pretty easy and probably more
> well-known than "decorators" (which I had never heard of, personally).

Yes, except function composition does not work on syntactic forms, which is why, with `compose`, you have to separate the `operating-system` form instead of being able to compose `operating-system` with the rest of the os modifications.

The intent is that you have already an existing `operating-system` form with a single layer of parenthesis. With `compose`, if you want to install ZFS and a bunch of other complex OS modifications, you have to add a layer of parenthesis.  With `decorate`, you don't:

```scheme
((compose install-zfs install-foo)
 (operating-system
    (name "my-system") #;...))
;vs
(decorate (install-zfs
           install-foo
           operating-system)
  (nmae "my-system") #;...)
```

>
> Fewer macros means the reader needs to keep fewer special rules in mind.

Fine, I'm doubling down then.

```scheme
(define-syntax compose-syntax
  (syntax-rules ()
    ((compose-syntax (x ...))
     (syntax-rules ::: ()
       ((form args :::)
        (x ... args :::))))
    ((compose-syntax x)
     (syntax-rules ::: ()
       ((form args :::)
        (x args :::))))
    ((compose-syntax (x ...) y ...)
     (syntax-rules ::: ()
       ((form args :::)
        (let-syntax ((sub-syntax (compose-syntax y ...)))
          (x ... (sub-syntax args :::))))))
    ((compose-syntax x y ...)
     (syntax-rules ::: ()
       ((form args :::)
        (let-syntax ((sub-syntax (compose-syntax y ...))
          (x (sub-syntax args :::)))))))))
```

Then use it as follows:

```
(define-syntax my-operating-system
  (compose-syntax
    (install-zfs #:options '(("zfs_arc_max" 5000000000)) #:os)
    operating-system))
(my-operating-system
  (name "my-system") #;...)
```

Again, the goal here is to keep the nestedness of your existing, very long `operating-system` form, which your simple `compose` function fails to do, because you can't compose syntax with `compose` and `operating-system` is a syntax form.

>
> Secondly, I wonder if passing an OS declaration through various
> procedures that modify it is really the best approach in the first place.
>
> For build phases, we have the 'modify-phases' syntax. For services,
> there is 'modify-services'. Maybe there should be a 'modify-os' kind of
> syntax. (In other words, if we're going to invent new syntax, why not
> go all-out and create the most convenient syntax for the use-case.)

Because a *generic* form to modify the operating system is worthless --- you can just define the operating-system directly.

What `install-zfs` does is that it installs the same kernel-specific package in three different points:

* `kernel-loadable-modules`, because ZFS needs to get into the kernel somehow.
* `packages`, because the kernel module is useless if you don't have the userland tools to interact with the kernel module.
* `services`, because ZFS is well-documented outside of Guix as automatically mounting its filesystems at bootup, but that actually requires a bit of magic in the `init` system, specifically you need to actually **load** the module, then execute `zpool import -a -l` to have it scan for all filesystems and mount those that need automounting.

Thus, an `install-zfs`, that is a *single* form that inserts the correct bits in the correct ***three*** places, makes the experience of adding ZFS to your `operating-system` easier because there's less scope for error in actually adding the package.  You just add a single `install-zfs`, not add three things (plus an extra `(define my-zfs (make-zfs-package linux-libre-5.4))` before your form).


Now, if this kind of simplifying form is useful for ZFS, I imagine that this kind of simplifying form would also exist for other things you could install into your operating system in the future.  Thus, we need some way to take an existing `<operating-system>` and pass it through a number of single simplifying operating system transformations, which I don't think something like `modify-os` would work well with.


Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: A new paradigm for modifying operating system declarations
  2021-01-04 23:26 ` Jan Wielkiewicz
@ 2021-01-05  0:46   ` raid5atemyhomework
  0 siblings, 0 replies; 7+ messages in thread
From: raid5atemyhomework @ 2021-01-05  0:46 UTC (permalink / raw)
  To: Jan Wielkiewicz; +Cc: guix-devel@gnu.org

Hi Jan,

>
> Better but still don't like it.
> Can't we put the os declaration into a variable and then pass it to a
> procedure?
> Say:
>
> >     (define OS
> >       (operating-system
> >         (kernel linux-libre-5.4)
> >         ; ... other fields ...
> >         ))
> >
> >     (install-zfs OS)
> >     (install-foo OS)
> >     (install-bar OS)
> >     (install-something OS)
> >
> >

No, because `<operating-system>` objects are not mutable (or at least their formal interface doesn't expose any mutation).

What we could do would be:

```scheme
(define os
  (operating-system
    (kernel linux-libre-5.4)
    #;...))

(set! os (install-zfs os))
(set! os (install-foo os))
(set! os (install-bar os))

os
```

However, in many examples I've seen, the `configuration.scm` file looks like this:

```scheme
(use-modules (gnu))
(use-package-modules #;...)
(use-service-modules #;...)

(operating-system
  (host-name "my-system")
  (timezone "Europe/Paris")
  (locale "en_US,utf-8")
  (kernel linux-libre-5.4)
  #;...)
```

What I want to avoid would be to have to nest the existing, usually screens-long, `operating-system` form.

So compare:

```scheme
(use-modules (gnu))
(use-package-modules #;...)
(use-service-modules #;...)

(decorate (install-zfs
           install-foo
           install-bar
           operating-system)
  (host-name "my-system")
  (timezone "Europe/Paris")
  (locale "en_US,utf-8")
  (kernel linux-libre-5.4)
  #;...)
```

versus:

```scheme
(use-modules (gnu))
(use-package-modules #;...)
(use-service-modules #;...)

(define os
  (operating-system
    (host-name "my-system")
    (timezone "Europe/Paris")
    (locale "en_US,utf-8")
    (kernel linux-libre-5.4)
    #;...))
(set! os (install-zfs os))
(set! os (install-foo os))
(set! os (install-bar os))
os
```

I feel the former is better and requires less boilerplate.

Now of course, I ***have*** seen examples as well where the `operating-system` is put in a `define` form as well, but those are rare and the default stuff that come with Guix tend not to use this, and we should consider that new Guix sysads might not be comfortable working with EMACS and prefer nano, so adding even just *one* additional layer of nestedness to a long `operating-system` form is not easy.  Of course, such a sysad might then consider not indenting it correctly.

Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: A new paradigm for modifying operating system declarations
  2021-01-05  0:32   ` raid5atemyhomework
@ 2021-01-05  6:12     ` Carlo Zancanaro
  2021-01-05 10:01       ` raid5atemyhomework
  0 siblings, 1 reply; 7+ messages in thread
From: Carlo Zancanaro @ 2021-01-05  6:12 UTC (permalink / raw)
  To: raid5atemyhomework; +Cc: guix-devel

Hi raid5atemyhomework,

On Tue, Jan 05 2021, raid5atemyhomework wrote:
> What `install-zfs` does is that it installs the same 
> kernel-specific package in three different points:
>
> * `kernel-loadable-modules`, because ZFS needs to get into the 
> kernel somehow.
> * `packages`, because the kernel module is useless if you don't 
> have the userland tools to interact with the kernel module.
> * `services`, because ZFS is well-documented outside of Guix as 
> automatically mounting its filesystems at bootup, but that 
> actually requires a bit of magic in the `init` system, 
> specifically you need to actually **load** the module, then 
> execute `zpool import -a -l` to have it scan for all filesystems 
> and mount those that need automounting.
>
> Thus, an `install-zfs`, that is a *single* form that inserts the 
> correct bits in the correct ***three*** places, makes the 
> experience of adding ZFS to your `operating-system` easier 
> because there's less scope for error in actually adding the 
> package.  You just add a single `install-zfs`, not add three 
> things (plus an extra `(define my-zfs (make-zfs-package 
> linux-libre-5.4))` before your form).

In principle, I think this should all be handled by a service. 
Services have a number of extension points where they can impact 
the operating system being declared, by extending other services. 
For example, adding a package into the global profile is done by 
extending profile-service-type (which you can find in 
gnu/services.scm). Adding a shepherd service to manage a process 
is done by extending shepherd-root-service-type (in 
gnu/services/shepherd.scm).

This is how many services work. As an example, sddm-service-type 
extends services to: (a) start a process on the running system, 
(b) put files in /etc, (c) install some pam services, (d) add an 
account on the system, and (e) install packages in the global 
profile.

As far as I can tell, the only thing missing for a 
zfs-service-type to do what you want is that services can't 
currently add new kernel modules (although they can load them via 
kernel-module-loader-service-type). I may have missed a mechanism 
for this, though. If we added the ability to do this, then it 
should be possible to add zfs support by adding a single (service 
zfs-service-type) to your services list.

The approach of using services in this way has some advantages 
which are outlined in a blog post from 2015[1]. For me the most 
compelling advantage is that an zfs-service-type is more 
restricted in what it can do, and must be more explicit. An 
install-zfs procedure has free-reign over the entire 
operating-system definition that it gets passed, which makes it 
harder to reason about the composition of such procedures.

Carlo

[1]: https://guix.gnu.org/blog/2015/service-composition-in-guixsd/


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: A new paradigm for modifying operating system declarations
  2021-01-05  6:12     ` Carlo Zancanaro
@ 2021-01-05 10:01       ` raid5atemyhomework
  0 siblings, 0 replies; 7+ messages in thread
From: raid5atemyhomework @ 2021-01-05 10:01 UTC (permalink / raw)
  To: Carlo Zancanaro; +Cc: guix-devel@gnu.org

Hi Carlo,

> In principle, I think this should all be handled by a service.
> Services have a number of extension points where they can impact
> the operating system being declared, by extending other services.
> For example, adding a package into the global profile is done by
> extending profile-service-type (which you can find in
> gnu/services.scm). Adding a shepherd service to manage a process
> is done by extending shepherd-root-service-type (in
> gnu/services/shepherd.scm).
>
> This is how many services work. As an example, sddm-service-type
> extends services to: (a) start a process on the running system,
> (b) put files in /etc, (c) install some pam services, (d) add an
> account on the system, and (e) install packages in the global
> profile.
>
> As far as I can tell, the only thing missing for a
> zfs-service-type to do what you want is that services can't
> currently add new kernel modules (although they can load them via
> kernel-module-loader-service-type). I may have missed a mechanism
> for this, though. If we added the ability to do this, then it
> should be possible to add zfs support by adding a single (service
> zfs-service-type) to your services list.
>
> The approach of using services in this way has some advantages
> which are outlined in a blog post from 2015[1]. For me the most
> compelling advantage is that an zfs-service-type is more
> restricted in what it can do, and must be more explicit. An
> install-zfs procedure has free-reign over the entire
> operating-system definition that it gets passed, which makes it
> harder to reason about the composition of such procedures.

Thank you for this, this is certainly something I would prefer to simplify installing ZFS onto the operating system!  I wasn't aware of the `profile-service-type`.

I already have a `zfs-loader-service-type` here: https://issues.guix.gnu.org/45643#1

These are the things that the service has to do:

* Get a kernel module compiled and added to the set of kernel modules that can be loaded.
  * From what I understand of what you describe, this is non-existent for now.  How difficult would this be?  Where do I start in getting this implemented?
* Load the kernel module.
* Execute `zpool import -a -l` to import all ZFS pools, mount those that are marked automount, and request for passwords if encrypted.
  * This has to occur after kernel module loading `(requirements '(kernel-module-loader))`.
  * This has to occur before `file-systems`.  In https://issues.guix.gnu.org/45643#2 I made `file-systems` into a target similar to `user-processes` (i.e. it has a `file-systems-service-type` that can be extended with a list of `requirements`, just like `user-procesess`) so that the `zpool import` service is a dependency of `file-systems`.
    * This is needed in order to get `/home` on ZFS, since `user-homes` can race with this and populate `/home` before the ZFS module is loaded and the pools are imported, and ZFS will refuse to mount on top of a non-empty directory.  We need to make sure that ZFS gets to mount before `file-systems` is started, since `file-systems` will also trigger `user-homes`.
* Add the compiled ZFS on the profile.
* Add ZFS module configuration into an `/etc/modprobe.d/zfs.conf`.

The above would not get us `/` on ZFS, since we would need to have the ZFS kernel module inside the `initrd`, have the kernel load the module before launch, probably import pools early, and then look for root pool and so on.  The problem is getting the ZFS configuration inside the `initrd` as well, in addition to the userspace tools.

So, here's a sketch:

```scheme
(operating-system
  (kernel linux-libre-5.4)
  ;...

  (services
    (cons* (service zfs-service-type
             (zfs-configuration
               ; we need to compile for a specific kernel, the alternative here
               ; would be (operating-system this-operating-system) but the below
               ; is a good bit shorter, despite the DRY violation...
               (kernel linux-libre-5.4)
               (options
                 '(("zfs_arc_max" 5000000000)))))
           ;...
           %desktop-services))

  #;...)
```

Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-01-05 10:01 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-04 15:38 A new paradigm for modifying operating system declarations raid5atemyhomework
2021-01-04 20:21 ` Taylan Kammer
2021-01-05  0:32   ` raid5atemyhomework
2021-01-05  6:12     ` Carlo Zancanaro
2021-01-05 10:01       ` raid5atemyhomework
2021-01-04 23:26 ` Jan Wielkiewicz
2021-01-05  0:46   ` raid5atemyhomework

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).