unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / Atom feed
* ZFS on Guix
@ 2021-01-02  6:16 raid5atemyhomework
  2021-01-02  6:40 ` raid5atemyhomework
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-02  6:16 UTC (permalink / raw)
  To: guix-devel

Hi guix-developers,

I'm trying to get ZFS working on a test VM running Guix System.

Attempting to do a simple `guix install zfs` does not work, as I reported in https://issues.guix.gnu.org/45401 , ZFS 0.8.5 is only up to kernel 5.9 but latest guix has kernel 5.10.

I created a procedure to modify the base `zfs` package and compile it for a different Linux kernel:

```scheme
(define (make-zfs-package kernel)
  (package
    (inherit zfs)
    (name (string-append "zfs-for-"
                         (package-name kernel)
                         "-"
                         (package-version kernel)
                         "-version"))
    (arguments
      (cons* #:linux kernel (package-arguments zfs)))))
```

Then in my system configuration `.scm`, I selected a specific Linux and had its module added to the boot and had it loaded in a Shepherd service:

```scheme
(define system-kernel linux-libre-5.4)
(define system-zfs (make-zfs-package system-kernel))

;; ...

(operating-system
  (kernel system-kernel)
  (kernel-loadable-modules (list (list system-zfs "module")))
  ;; ...
  (packages
    (cons* system-zfs
           ; ...
           %base-packages))
  ;; ...

  (services
    (cons* (service kernel-module-loader-service-type
                    '("zfs"))
           ; ...
           %base-services))
  ;; ...
  )
```

As I documented here: https://issues.guix.gnu.org/45592#1

Now ZFS is loaded at boot (`lsmod | grep zfs` shows ZFS loaded), and `zfs version` and `zpool list` don't error out.

However, when I created some extra space in the QEMU image and created a partition `vda3`, I get an error in `zpool create tank vda3`.
The error is `cannot mount 'tank': Input/output error`.

I've also tried adding a new device to the QEMU boot as well, also the same occurs.

Looking at a partition editor, the partition does get formatted as zfs, it just won't mount.



Has anyone ever managed to get ZFS working on Guix?  Any tips?  Is it enough to `modprobe` it in a system service or does it have to somehow be added to the Linux command line?  Maybe it only works on even older Linux-libre versions than 5.4?

Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-02  6:16 ZFS on Guix raid5atemyhomework
@ 2021-01-02  6:40 ` raid5atemyhomework
  2021-01-03 15:50   ` Danny Milosavljevic
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-02  6:40 UTC (permalink / raw)
  To: guix-devel

Hi guix-developers,

I just did a an `strace -f zfs mount tank`, and got this interesting bit in a child process:

    execve("/gnu/store/a45p39mgqvfd8kjwibyr0q42klmw7gmf-util-linux-2.35.1-lib/bin/mount", ["/gnu/store/a45p39mgqvfd8kjwibyr0"..., "--no-canonicalize", "-t", "zfs", "-o", "defaults,atime,strictatime,dev,e"..., "tank", "/tank"], 0x7ffee6ad7bf8 /* 29 vars */) = -1 ENOENT (No such file or directory)

The `/gnu/store` directory indeed does not contain a `/bin/mount` executable, it only contains `include` `lib` and `share` dirs.  So it looks to me that maybe the `zfs` package needs some more modifications to get it working on guix, since apparently ZFS userspace tools still go through `mount` anyway.

Trying to `mount` directly makes ZFS complain about requiring a "legacy" mountpoint (there might be an option somewhere not shown by the `strace` that suppress that complaint), and setting a legacy mountpoint makes `mount` error with `filesystem 'tank' cannot be mounted at '/tank' due to canonicalization error 2.` even though I give `--no-canonicalize`, though.

Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-02  6:40 ` raid5atemyhomework
@ 2021-01-03 15:50   ` Danny Milosavljevic
       [not found]     ` <Kyzl4xnRrDtWbTkAmBf1i8mtSZvu-AYatdazY1NsABFAzqqi7HQl-t0d2LWAInr8n7KxGyJWfIfTWqrefrxdgWelKbr2SWc9ATV5P8zrVtw=3D@protonmail.com>
  2021-01-04  0:50     ` raid5atemyhomework
  0 siblings, 2 replies; 15+ messages in thread
From: Danny Milosavljevic @ 2021-01-03 15:50 UTC (permalink / raw)
  To: raid5atemyhomework; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 866 bytes --]

Hi,

the reason is that our "zfs" package uses

  ("util-linux" ,util-linux "lib")

and then does

               (substitute* "lib/libzfs/libzfs_mount.c"
                 (("/bin/mount") (string-append util-linux "/bin/mount"))
                 (("/bin/umount") (string-append util-linux "/bin/umount")))

.

That can't work.

zfs interna that will be patched by the Guix package are:

./lib/libzfs/libzfs_mount.c

do_mount(const char *src, const char *mntpt, char *opts)

        char *argv[9] = {
            "/bin/mount",
            "--no-canonicalize",
            "-t", MNTTYPE_ZFS,
            "-o", opts,
            (char *)src,
            (char *)mntpt,
            (char *)NULL };

Easy fix would be to change our package to have both

  ("util-linux:lib" ,util-linux "lib")

and

  ("util-linux" ,util-linux)

.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-03 15:50   ` Danny Milosavljevic
       [not found]     ` <Kyzl4xnRrDtWbTkAmBf1i8mtSZvu-AYatdazY1NsABFAzqqi7HQl-t0d2LWAInr8n7KxGyJWfIfTWqrefrxdgWelKbr2SWc9ATV5P8zrVtw=3D@protonmail.com>
@ 2021-01-04  0:50     ` raid5atemyhomework
  2021-01-04  1:15       ` raid5atemyhomework
  1 sibling, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-04  0:50 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

Yes, I found that out after a little more digging.

I have a preliminary patch here: https://issues.guix.gnu.org/45592
However I will make a patchset based on this in order to support /home on ZFS and managing ZFS the "legacy" way by (file-system ...) declaration, please wait.



> Hi,
>
> the reason is that our "zfs" package uses
>
> ("util-linux" ,util-linux "lib")
>
> and then does
>
> (substitute* "lib/libzfs/libzfs_mount.c"
> (("/bin/mount") (string-append util-linux "/bin/mount"))
> (("/bin/umount") (string-append util-linux "/bin/umount")))
>
> .
>
> That can't work.
>
> zfs interna that will be patched by the Guix package are:
>
> ./lib/libzfs/libzfs_mount.c
>
> do_mount(const char *src, const char *mntpt, char *opts)
>
> char *argv[9] = {
> "/bin/mount",
> "--no-canonicalize",
> "-t", MNTTYPE_ZFS,
> "-o", opts,
> (char *)src,
> (char *)mntpt,
> (char *)NULL };
>
> Easy fix would be to change our package to have both
>
> ("util-linux:lib" ,util-linux "lib")
>
> and
>
> ("util-linux" ,util-linux)
>
> .




^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-04  0:50     ` raid5atemyhomework
@ 2021-01-04  1:15       ` raid5atemyhomework
  2021-01-05 11:02         ` raid5atemyhomework
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-04  1:15 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

Available here: https://issues.guix.gnu.org/45643

Please review! Hopefully it gets merged!

> Yes, I found that out after a little more digging.
>
> I have a preliminary patch here: https://issues.guix.gnu.org/45592
> However I will make a patchset based on this in order to support /home on ZFS and managing ZFS the "legacy" way by (file-system ...) declaration, please wait.
>
> > Hi,
> > the reason is that our "zfs" package uses
> > ("util-linux" ,util-linux "lib")
> > and then does
> > (substitute* "lib/libzfs/libzfs_mount.c"
> > (("/bin/mount") (string-append util-linux "/bin/mount"))
> > (("/bin/umount") (string-append util-linux "/bin/umount")))
> > .
> > That can't work.
> > zfs interna that will be patched by the Guix package are:
> > ./lib/libzfs/libzfs_mount.c
> > do_mount(const char *src, const char *mntpt, char *opts)
> > char *argv[9] = {
> > "/bin/mount",
> > "--no-canonicalize",
> > "-t", MNTTYPE_ZFS,
> > "-o", opts,
> > (char *)src,
> > (char *)mntpt,
> > (char *)NULL };
> > Easy fix would be to change our package to have both
> > ("util-linux:lib" ,util-linux "lib")
> > and
> > ("util-linux" ,util-linux)
> > .




^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-04  1:15       ` raid5atemyhomework
@ 2021-01-05 11:02         ` raid5atemyhomework
  2021-01-05 14:56           ` raid5atemyhomework
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-05 11:02 UTC (permalink / raw)
  To: Carlo Zancanaro; +Cc: guix-devel

Hi guix-developers,


In https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00053.html Carlo mentioned to instead use the `service` system to modify all parts needed in the operating system to get ZFS installed, and noted as well that the only thing missing in the `service` system is the ability to push kernel modules into the operating system.

Here's a sketch of an attempt to add that feature so we can conveniently add ZFS on Guix by just adding a `(service zfs-service-type...)`.

First, we need a new `kernel-loadable-module-service-type` in `gnu/services.scm`:

```scheme
(define kernel-loadable-module-service-type
  ;; A service to add the kernel modules of a package.
  (service-type (name 'kernel-loadable-module)
                (extensions '())
                (compose concatenate)
                (extend append)
                (description "Register kernel modules that will be placed in a
loadable place where the kernel can find them.")
                (default-value '())))
```

Absolutely no idea if a `'()` extensions makes sense but *shrug*.

Then, in `gnu/system.scm`, we create this new function:

```scheme
(define (operating-system-all-kernel-loadable-modules os)
  ;; Gathers the kernel-loadable modules in the KERNEL-LOADABLE-FIELD
  ;; field of the OS, as well as those registered by services.
  (let ((service (fold-services (cons (service kernel-loadable-module-service-type '())
                                      (operating-system-user-services os))
                                #:target-type kernel-loadable-module-service-type)))
    (append (service-value service)
            (operating-system-kernel-loadable-modules))))
```

Does that make sense? Am I misusing `fold-services` and `service-value` here?

Onward, we modify as well the function `operating-system-directory-base`:

```scheme
(define* (operating-system-directory-base-entries os)
  "Return the basic entries of the 'system' directory of OS for use as the
value of the SYSTEM-SERVICE-TYPE service."
  (let* ((locale  (operating-system-locale-directory os))
         (kernel  (operating-system-kernel os))
         (hurd    (operating-system-hurd os))
         (modules (operating-system-all-kernel-loadable-modules os))
;...
```

The above is the reason why we need to `cons` a synthetic `kernel-loadable-module-service-type` service, the function `operating-system-directory-base-entries` is used do create the default for `operating-system-essential-services`, and the `operating-sstem-services` is just the concatenation of the user and essential services.  I think.

Would this work?

Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-05 11:02         ` raid5atemyhomework
@ 2021-01-05 14:56           ` raid5atemyhomework
  2021-01-06  0:59             ` Carlo Zancanaro
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-05 14:56 UTC (permalink / raw)
  To: Carlo Zancanaro; +Cc: guix-devel

Hi Carlo and guix-developers,

I have this patch below for creating a new `kernel-loadable-module-service-type`, which can be extended by another service to add kernel-loadable modules provided by packages, hope for a review.


From 984602faba1e18b9eb64e62970147aab653d997f Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH] gnu: Add 'kernel-loadable-module-service-type' for services
 to extend with kernel-loadable modules.

---
 doc/guix.texi    |  6 +++++
 gnu/services.scm | 68 ++++++++++++++++++++++++++++++++++++++++++++++++
 gnu/system.scm   | 36 +++++++------------------
 3 files changed, 83 insertions(+), 27 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 0f6e95a65a..78770151e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32409,6 +32409,12 @@ configuration when you use @command{guix system reconfigure},
 @command{guix system init}, or @command{guix deploy}.
 @end defvr

+@defvr {Scheme Variable} kernel-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+@end defvr
+
 @node Shepherd Services
 @subsection Shepherd Services

diff --git a/gnu/services.scm b/gnu/services.scm
index 13259dfaee..2c0bbf9725 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -33,6 +33,8 @@
   #:use-module (guix diagnostics)
   #:autoload   (guix openpgp) (openpgp-format-fingerprint)
   #:use-module (guix modules)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
   #:use-module (gnu packages base)
   #:use-module (gnu packages bash)
   #:use-module (gnu packages hurd)
@@ -75,6 +77,7 @@
             service-back-edges
             instantiate-missing-services
             fold-services
+            kernel-loadable-module-service

             service-error?
             missing-value-service-error?
@@ -106,6 +109,7 @@
             profile-service-type
             firmware-service-type
             gc-root-service-type
+            kernel-loadable-module-service-type

             %boot-service
             %activation-service
@@ -864,6 +868,70 @@ as Wifi cards.")))
 will not be reclaimed by the garbage collector.")
                 (default-value '())))

+;; Configuration for the kernel builder.
+;; Only used by the KERNEL-LOADABLE-MODULE-SERVICE-TYPE.
+(define-record-type* <kernel-builder-configuration> kernel-builder-configuration
+  make-kernel-builder-configuration
+  kernel-builder-configuration?
+  this-kernel-builder-configuration
+
+  (kernel   kernel-builder-configuration-kernel   (default #f))
+  (hurd     kernel-builder-configuration-hurd     (default #f))
+  (modules  kernel-builder-configuration-modules  (default '())))
+
+(define (package-for-kernel target-kernel module-package)
+  "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
+possible (that is if there's a LINUX keyword argument in the build system)."
+  (package
+    (inherit module-package)
+    (arguments
+     (substitute-keyword-arguments (package-arguments module-package)
+       ((#:linux kernel #f)
+        target-kernel)))))
+
+(define (kernel-builder-configuration->system-entry config)
+  "Return the kernel and hurd entries of the 'system' directory of OS."
+  (mbegin %store-monad
+    (let* ((kernel  (kernel-builder-configuration-kernel config))
+           (hurd    (kernel-builder-configuration-hurd config))
+           (modules (kernel-builder-configuration-modules config))
+           (kernel  (if hurd
+                        kernel
+                        (profile
+                         (content (packages->manifest
+                                   (cons kernel
+                                         (map (lambda (module)
+                                                (if (package? module)
+                                                    (package-for-kernel kernel module)
+                                                    module))
+                                              modules))))
+                         (hooks (list linux-module-database))))))
+      (return `(("kernel" ,kernel)
+                ,@(if hurd `(("hurd" ,hurd)) '()))))))
+(define (kernel-builder-configuration-add-modules config modules)
+  "Constructs a kernel builder configuration that has its modules extended."
+  (kernel-builder-configuration
+    (inherit config)
+    (modules (append (kernel-builder-configuration-modules config) modules))))
+
+(define kernel-loadable-module-service-type
+  (service-type (name 'kernel-loadable-modules)
+                (extensions
+                 (list (service-extension system-service-type
+                                          kernel-builder-configuration->system-entry)))
+                (compose concatenate)
+                (extend kernel-builder-configuration-add-modules)
+                (description
+                 "Register packages containing kernel-loadable modules and adds them
+to the system.")))
+(define (kernel-loadable-module-service kernel hurd modules)
+  "Constructs the service that sets up kernel loadable modules."
+  (service kernel-loadable-module-service-type
+    (kernel-builder-configuration
+      (kernel kernel)
+      (hurd hurd)
+      (modules modules))))
+
 \f
 ;;;
 ;;; Service folding.
diff --git a/gnu/system.scm b/gnu/system.scm
index c284a18379..a7f2365754 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -600,16 +600,6 @@ OS."
       (file-append (operating-system-kernel os)
                       "/" (system-linux-image-file-name))))

-(define (package-for-kernel target-kernel module-package)
-  "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
-possible (that is if there's a LINUX keyword argument in the build system)."
-  (package
-    (inherit module-package)
-    (arguments
-     (substitute-keyword-arguments (package-arguments module-package)
-       ((#:linux kernel #f)
-        target-kernel)))))
-
 (define %default-modprobe-blacklist
   ;; List of kernel modules to blacklist by default.
   '("usbmouse" ;races with bcm5974, see <https://bugs.gnu.org/35574>
@@ -625,26 +615,10 @@ possible (that is if there's a LINUX keyword argument in the build system)."
   "Return the basic entries of the 'system' directory of OS for use as the
 value of the SYSTEM-SERVICE-TYPE service."
   (let* ((locale  (operating-system-locale-directory os))
-         (kernel  (operating-system-kernel os))
          (hurd    (operating-system-hurd os))
-         (modules (operating-system-kernel-loadable-modules os))
-         (kernel  (if hurd
-                      kernel
-                      (profile
-                       (content (packages->manifest
-                                 (cons kernel
-                                       (map (lambda (module)
-                                              (if (package? module)
-                                                  (package-for-kernel kernel
-                                                                      module)
-                                                  module))
-                                            modules))))
-                       (hooks (list linux-module-database)))))
          (initrd  (and (not hurd) (operating-system-initrd-file os)))
          (params  (operating-system-boot-parameters-file os)))
-    `(("kernel" ,kernel)
-      ,@(if hurd `(("hurd" ,hurd)) '())
-      ("parameters" ,params)
+    `(("parameters" ,params)
       ,@(if initrd `(("initrd" ,initrd)) '())
       ("locale" ,locale))))   ;used by libc

@@ -663,6 +637,10 @@ bookkeeping."
          (host-name (host-name-service (operating-system-host-name os)))
          (entries   (operating-system-directory-base-entries os)))
     (cons* (service system-service-type entries)
+           (kernel-loadable-module-service
+             (operating-system-kernel os)
+             (operating-system-hurd os)
+             (operating-system-kernel-loadable-modules os))
            %boot-service

            ;; %SHEPHERD-ROOT-SERVICE must come last so that the gexp that
@@ -699,6 +677,10 @@ bookkeeping."
 (define (hurd-default-essential-services os)
   (let ((entries (operating-system-directory-base-entries os)))
     (list (service system-service-type entries)
+          (kernel-loadable-module-service
+            (operating-system-kernel os)
+            (operating-system-hurd os)
+            (operating-system-kernel-loadable-modules os))
           %boot-service
           %hurd-startup-service
           %activation-service
--
2.29.2



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-05 14:56           ` raid5atemyhomework
@ 2021-01-06  0:59             ` Carlo Zancanaro
  2021-01-06  3:50               ` raid5atemyhomework
  0 siblings, 1 reply; 15+ messages in thread
From: Carlo Zancanaro @ 2021-01-06  0:59 UTC (permalink / raw)
  To: raid5atemyhomework; +Cc: guix-devel

Hi raid5atemyhomework,

On Wed, Jan 06 2021, raid5atemyhomework wrote:
> I have this patch below for creating a new 
> `kernel-loadable-module-service-type`, which can be extended by 
> another service to add kernel-loadable modules provided by 
> packages, hope for a review.

I tried out building a VM using this patch and it seemed to work 
properly. I was able to add a kernel module by extending 
kernel-loadable-module-service-type, and then load it in the 
running VM with modprobe.

I only have superficial comments about the patch itself. Hopefully 
someone else will provide a better review than I can.

> From 984602faba1e18b9eb64e62970147aab653d997f Mon Sep 17 
> 00:00:00 2001
> From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
> Date: Tue, 5 Jan 2021 22:27:56 +0800
> Subject: [PATCH] gnu: Add 'kernel-loadable-module-service-type' 
> for services
>  to extend with kernel-loadable modules.

This will need a proper commit message before being committed. 
Guix generally follows the GNU Change Log style[1]. You can see 
examples of what commit messages look like by looking at other 
commits in the repository.

> +;; Only used by the KERNEL-LOADABLE-MODULE-SERVICE-TYPE.

I would remove this comment. This is a description of the current 
state, rather than a restriction on how it should be used. As a 
description, it might get out of date (if we use 
<kernel-builder-configuration> values elsewhere), and it doesn't 
add much value (because it can be easily verified, and the record 
type isn't exported). If you indent this to be normative, the 
comment should explain why this type should not be used elsewhere.

> +      (return `(("kernel" ,kernel)
> +                ,@(if hurd `(("hurd" ,hurd)) '()))))))
> +(define (kernel-builder-configuration-add-modules config 
> modules)
> +  "Constructs a kernel builder configuration that has its 
> modules extended."

Put empty lines between definitions here ...

> +                 "Register packages containing kernel-loadable 
> modules and adds them
> +to the system.")))
> +(define (kernel-loadable-module-service kernel hurd modules)
> +  "Constructs the service that sets up kernel loadable 
> modules."

... and here.

It would also be worth adding a test to 
gnu/tests/linux-modules.scm to test loading modules through this 
service.

Other than that, it looks good to me and it seems to work in my 
limited testing. 👍 Good job!

Carlo

[1]: https://www.gnu.org/prep/standards/html_node/Change-Logs.html


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-06  0:59             ` Carlo Zancanaro
@ 2021-01-06  3:50               ` raid5atemyhomework
  2021-01-06  3:58                 ` Carlo Zancanaro
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-06  3:50 UTC (permalink / raw)
  To: Carlo Zancanaro; +Cc: guix-devel

Hi Carlo,

Thanks, I modified patch as below.

I also can't get `guix system vm` to run in my hacking env, so I couldn't make a VM for testing like you did.  I got this:

```
Formatting '/gnu/store/imdlq0cay61d2cw3199g04z9bp16qx8j-qemu-image', fmt=qcow2 cluster_size=65536 compression_type=zlib size=73400320 lazy_refcounts=off refcount_bits=16
Could not access KVM kernel module: Permission denied
qemu-system-x86_64: failed to initialize kvm: Permission denied
```

Any tips on getting `guix system vm` working?  I've never used VMs before.  My dev env is a foreign distro with Guix installed on top.

The above issue also means I haven't actually run the tests I made in `gnu/tests/linux-modules.scm`, because it requires building a VM as well.

However, I *did* do a bunch of `guix system build` tests:

* With the same `configuration.scm` that uses the legacy `(kernel-loadable-modules ...)` declaration in `operating-system`, I tried both with and without the patch.
  * Without patch and with patch created builds with a different hash, however, they linked to the same directories. `diff -r` did not find any differences.
* With the patch, I made two variants of `configuration.scm`, one that used the legacy `(kernel-loadable-modules...)` and another that used a service that extended `kernel-laodable-module-service-type`.
  * Builds resulted in the exact same system with exact same hash.



From 4beb73c62995cf236b402dad8e1c36016027c781 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH] gnu: Allow services to install kernel-loadable modules.

* gnu/system.scm (operating-system-directory-base-entries): Remove code
to handle generation of "kernel" and "hurd".
(operating-system-default-essential-services): Instantiate
kernel-loadable-module-service.
(hurd-default-essential-services): Instantiate
kernel-loadable-module-service.
(package-for-kernel): Move ...
* gnu/services.scm: ... to here.
(kernel-loadable-module-service-type): New variable.
(kernel-loadable-module-service): New procedure.
* gnu/tests/linux-modules.scm (run-loadable-kernel-modules-test): Move
code to ...
(run-loadable-kernel-modules-test-base): ... new procedure here.
(run-loadable-kernel-modules-service-test): New procedure.
(%test-loadable-kernel-modules-service-0): New variable.
(%test-loadable-kernel-modules-service-1): New variable.
(%test-loadable-kernel-modules-service-2): New variable.
* doc/guix.texi: Document kernel-loadable-module-service-type.
---
 doc/guix.texi               |  6 +++
 gnu/services.scm            | 70 ++++++++++++++++++++++++++++++++
 gnu/system.scm              | 37 +++++------------
 gnu/tests/linux-modules.scm | 81 ++++++++++++++++++++++++++++++++-----
 4 files changed, 157 insertions(+), 37 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 0f6e95a65a..78770151e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32409,6 +32409,12 @@ configuration when you use @command{guix system reconfigure},
 @command{guix system init}, or @command{guix deploy}.
 @end defvr

+@defvr {Scheme Variable} kernel-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+@end defvr
+
 @node Shepherd Services
 @subsection Shepherd Services

diff --git a/gnu/services.scm b/gnu/services.scm
index 13259dfaee..d7332a46b2 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2016 Chris Marusich <cmmarusich@gmail.com>
 ;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -33,6 +34,8 @@
   #:use-module (guix diagnostics)
   #:autoload   (guix openpgp) (openpgp-format-fingerprint)
   #:use-module (guix modules)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
   #:use-module (gnu packages base)
   #:use-module (gnu packages bash)
   #:use-module (gnu packages hurd)
@@ -75,6 +78,7 @@
             service-back-edges
             instantiate-missing-services
             fold-services
+            kernel-loadable-module-service

             service-error?
             missing-value-service-error?
@@ -106,6 +110,7 @@
             profile-service-type
             firmware-service-type
             gc-root-service-type
+            kernel-loadable-module-service-type

             %boot-service
             %activation-service
@@ -864,6 +869,71 @@ as Wifi cards.")))
 will not be reclaimed by the garbage collector.")
                 (default-value '())))

+;; Configuration for the kernel builder.
+(define-record-type* <kernel-builder-configuration> kernel-builder-configuration
+  make-kernel-builder-configuration
+  kernel-builder-configuration?
+  this-kernel-builder-configuration
+
+  (kernel   kernel-builder-configuration-kernel   (default #f))
+  (hurd     kernel-builder-configuration-hurd     (default #f))
+  (modules  kernel-builder-configuration-modules  (default '())))
+
+(define (package-for-kernel target-kernel module-package)
+  "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
+possible (that is if there's a LINUX keyword argument in the build system)."
+  (package
+    (inherit module-package)
+    (arguments
+     (substitute-keyword-arguments (package-arguments module-package)
+       ((#:linux kernel #f)
+        target-kernel)))))
+
+(define (kernel-builder-configuration->system-entry config)
+  "Return the kernel and hurd entries of the 'system' directory."
+  (mbegin %store-monad
+    (let* ((kernel  (kernel-builder-configuration-kernel config))
+           (hurd    (kernel-builder-configuration-hurd config))
+           (modules (kernel-builder-configuration-modules config))
+           (kernel  (if hurd
+                        kernel
+                        (profile
+                         (content (packages->manifest
+                                   (cons kernel
+                                         (map (lambda (module)
+                                                (if (package? module)
+                                                    (package-for-kernel kernel module)
+                                                    module))
+                                              modules))))
+                         (hooks (list linux-module-database))))))
+      (return `(("kernel" ,kernel)
+                ,@(if hurd `(("hurd" ,hurd)) '()))))))
+
+(define (kernel-builder-configuration-add-modules config modules)
+  "Constructs a kernel builder configuration that has its modules extended."
+  (kernel-builder-configuration
+    (inherit config)
+    (modules (append (kernel-builder-configuration-modules config) modules))))
+
+(define kernel-loadable-module-service-type
+  (service-type (name 'kernel-loadable-modules)
+                (extensions
+                 (list (service-extension system-service-type
+                                          kernel-builder-configuration->system-entry)))
+                (compose concatenate)
+                (extend kernel-builder-configuration-add-modules)
+                (description
+                 "Register packages containing kernel-loadable modules and adds them
+to the system.")))
+
+(define (kernel-loadable-module-service kernel hurd modules)
+  "Constructs the service that sets up kernel loadable modules."
+  (service kernel-loadable-module-service-type
+    (kernel-builder-configuration
+      (kernel kernel)
+      (hurd hurd)
+      (modules modules))))
+
 \f
 ;;;
 ;;; Service folding.
diff --git a/gnu/system.scm b/gnu/system.scm
index c284a18379..5c530f176e 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -12,6 +12,7 @@
 ;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <jannek@gnu.org>
 ;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -600,16 +601,6 @@ OS."
       (file-append (operating-system-kernel os)
                       "/" (system-linux-image-file-name))))

-(define (package-for-kernel target-kernel module-package)
-  "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
-possible (that is if there's a LINUX keyword argument in the build system)."
-  (package
-    (inherit module-package)
-    (arguments
-     (substitute-keyword-arguments (package-arguments module-package)
-       ((#:linux kernel #f)
-        target-kernel)))))
-
 (define %default-modprobe-blacklist
   ;; List of kernel modules to blacklist by default.
   '("usbmouse" ;races with bcm5974, see <https://bugs.gnu.org/35574>
@@ -625,26 +616,10 @@ possible (that is if there's a LINUX keyword argument in the build system)."
   "Return the basic entries of the 'system' directory of OS for use as the
 value of the SYSTEM-SERVICE-TYPE service."
   (let* ((locale  (operating-system-locale-directory os))
-         (kernel  (operating-system-kernel os))
          (hurd    (operating-system-hurd os))
-         (modules (operating-system-kernel-loadable-modules os))
-         (kernel  (if hurd
-                      kernel
-                      (profile
-                       (content (packages->manifest
-                                 (cons kernel
-                                       (map (lambda (module)
-                                              (if (package? module)
-                                                  (package-for-kernel kernel
-                                                                      module)
-                                                  module))
-                                            modules))))
-                       (hooks (list linux-module-database)))))
          (initrd  (and (not hurd) (operating-system-initrd-file os)))
          (params  (operating-system-boot-parameters-file os)))
-    `(("kernel" ,kernel)
-      ,@(if hurd `(("hurd" ,hurd)) '())
-      ("parameters" ,params)
+    `(("parameters" ,params)
       ,@(if initrd `(("initrd" ,initrd)) '())
       ("locale" ,locale))))   ;used by libc

@@ -663,6 +638,10 @@ bookkeeping."
          (host-name (host-name-service (operating-system-host-name os)))
          (entries   (operating-system-directory-base-entries os)))
     (cons* (service system-service-type entries)
+           (kernel-loadable-module-service
+             (operating-system-kernel os)
+             (operating-system-hurd os)
+             (operating-system-kernel-loadable-modules os))
            %boot-service

            ;; %SHEPHERD-ROOT-SERVICE must come last so that the gexp that
@@ -699,6 +678,10 @@ bookkeeping."
 (define (hurd-default-essential-services os)
   (let ((entries (operating-system-directory-base-entries os)))
     (list (service system-service-type entries)
+          (kernel-loadable-module-service
+            (operating-system-kernel os)
+            (operating-system-hurd os)
+            (operating-system-kernel-loadable-modules os))
           %boot-service
           %hurd-startup-service
           %activation-service
diff --git a/gnu/tests/linux-modules.scm b/gnu/tests/linux-modules.scm
index 953b132ef7..9739e4124d 100644
--- a/gnu/tests/linux-modules.scm
+++ b/gnu/tests/linux-modules.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2019 Jakob L. Kreuze <zerodaysfordays@sdf.org>
 ;;; Copyright © 2020 Danny Milosavljevic <dannym@scratchpost.org>
 ;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -34,7 +35,10 @@
   #:use-module (guix utils)
   #:export (%test-loadable-kernel-modules-0
             %test-loadable-kernel-modules-1
-            %test-loadable-kernel-modules-2))
+            %test-loadable-kernel-modules-2
+            %test-loadable-kernel-modules-service-0
+            %test-loadable-kernel-modules-service-1
+            %test-loadable-kernel-modules-service-2))

 ;;; Commentary:
 ;;;
@@ -66,17 +70,11 @@ that MODULES are actually loaded."
                        (member module modules string=?))
                      '#$modules))))))

-(define* (run-loadable-kernel-modules-test module-packages module-names)
-  "Run a test of an OS having MODULE-PACKAGES, and verify that MODULE-NAMES
-are loaded in memory."
+(define* (run-loadable-kernel-modules-test-base base-os module-names)
+  "Run a test of BASE-OS, verifying that MODULE-NAMES are loaded in memory."
   (define os
     (marionette-operating-system
-     (operating-system
-      (inherit (simple-operating-system))
-      (services (cons (service kernel-module-loader-service-type module-names)
-                      (operating-system-user-services
-                       (simple-operating-system))))
-      (kernel-loadable-modules module-packages))
+     base-os
      #:imported-modules '((guix combinators))))
   (define vm (virtual-machine os))
   (define (test script)
@@ -98,6 +96,37 @@ are loaded in memory."
   (gexp->derivation "loadable-kernel-modules"
                     (test (modules-loaded?-program os module-names))))

+(define* (run-loadable-kernel-modules-test module-packages module-names)
+  "Run a test of an OS having MODULE-PACKAGES, and verify that MODULE-NAMES
+are loaded in memory."
+  (run-loadable-kernel-modules-test-base
+    (operating-system
+      (inherit (simple-operating-system))
+      (services (cons (service kernel-module-loader-service-type module-names)
+                      (operating-system-user-services
+                       (simple-operating-system))))
+      (kernel-loadable-modules module-packages))
+    module-names))
+
+(define* (run-loadable-kernel-modules-service-test module-packages module-names)
+  "Run a test of an OS having MODULE-PACKAGES, which are loaded by creating a
+service that extends KERNEL-LOADABLE-MODULE-SERVICE-TYPE. Then verify that
+MODULE-NAMES are loaded in memory."
+  (define module-installing-service-type
+    (service-type
+      (name 'module-installing-service)
+      (extensions (list (service-extension kernel-loadable-module-service-type
+                                           (const module-packages))))
+      (default-value #f)))
+  (run-loadable-kernel-modules-test-base
+    (operating-system
+      (inherit (simple-operating-system))
+      (services (cons* (service kernel-module-loader-service-type module-names)
+                       (service module-installing-service-type)
+                       (operating-system-user-services
+                        (simple-operating-system)))))
+    module-names))
+
 (define %test-loadable-kernel-modules-0
   (system-test
    (name "loadable-kernel-modules-0")
@@ -129,3 +158,35 @@ with two extra modules.")
                                                  (package-arguments
                                                   ddcci-driver-linux))))))
            '("acpi_call" "ddcci")))))
+
+(define %test-loadable-kernel-modules-service-0
+  (system-test
+   (name "loadable-kernel-modules-service-0")
+   (description "Tests loadable kernel modules extensible service with no
+extra modules.")
+   (value (run-loadable-kernel-modules-service-test '() '()))))
+
+(define %test-loadable-kernel-modules-service-1
+  (system-test
+   (name "loadable-kernel-modules-service-1")
+   (description "Tests loadable kernel modules extensible service with one
+extra module.")
+   (value (run-loadable-kernel-modules-service-test
+           (list ddcci-driver-linux)
+           '("ddcci")))))
+
+(define %test-loadable-kernel-modules-service-2
+  (system-test
+   (name "loadable-kernel-modules-service-2")
+   (description "Tests loadable kernel modules extensible service with two
+extra modules.")
+   (value (run-loadable-kernel-modules-service-test
+           (list acpi-call-linux-module
+                 (package
+                   (inherit ddcci-driver-linux)
+                   (arguments
+                    `(#:linux #f
+                      ,@(strip-keyword-arguments '(#:linux)
+                                                 (package-arguments
+                                                  ddcci-driver-linux))))))
+           '("acpi_call" "ddcci")))))
--
2.29.2



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-06  3:50               ` raid5atemyhomework
@ 2021-01-06  3:58                 ` Carlo Zancanaro
       [not found]                   ` <t=5Ft9LHglTTJw2Bhtd2xX4JCJeZBi5Drqg0t04vJsfRLfoENqpftZcuHO8LYi2AO05P71lbbCgQC5etCDiPsdcssJmw1pHGyCY3 gUT-9w9?= =?us-ascii?Q?=5Fo=3D@protonmail.com>
  2021-01-06  4:41                   ` raid5atemyhomework
  0 siblings, 2 replies; 15+ messages in thread
From: Carlo Zancanaro @ 2021-01-06  3:58 UTC (permalink / raw)
  To: raid5atemyhomework; +Cc: guix-devel

Apologies for the short reply, but that permission issue can be solved in the short term by changing the permissions on /dev/kvm (I set it to 777 to force it when necessary, because I don't know what the right permissions are).

The proper solution is probably to add your user to a kvm group on your distribution, which should give you the appropriate access.


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-06  3:58                 ` Carlo Zancanaro
       [not found]                   ` <t=5Ft9LHglTTJw2Bhtd2xX4JCJeZBi5Drqg0t04vJsfRLfoENqpftZcuHO8LYi2AO05P71lbbCgQC5etCDiPsdcssJmw1pHGyCY3 gUT-9w9?= =?us-ascii?Q?=5Fo=3D@protonmail.com>
@ 2021-01-06  4:41                   ` raid5atemyhomework
  2021-01-06  5:20                     ` raid5atemyhomework
  1 sibling, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-06  4:41 UTC (permalink / raw)
  To: Carlo Zancanaro; +Cc: guix-devel

Thank you Carlo!

I've now tested the new tests I added in `gnu/tests/linux-modules.scm`, and the existing tests as well, and they all pass.

Hope to get some review on that patch!

Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-06  4:41                   ` raid5atemyhomework
@ 2021-01-06  5:20                     ` raid5atemyhomework
  2021-01-06 15:58                       ` raid5atemyhomework
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-06  5:20 UTC (permalink / raw)
  To: guix-devel

Hi guix-developers,

Another issue here is that ZFS prefers to not be managed via `/etc/fstab`/`mount`/`umount`.  Instead, at startup ZFS magically imports ZFS pools and mounts ZFS datasets in the correct place, as configured in the `mountpoint` properties of the dataset.  This magic is actually implemented by executing `zpool import -a` at startup after ZFS module loading.  Note that `zpool import -a` has to be executed always so that ZFS can detect its pools, though the automatic mounting can be suppressed with `zpool import -a -N`.

(`systemd`-based distros have a number of `systemd` services that handle ZFS importation by use of a `/etc/zfs/zpool.cache` file that ZFS maintains, and then does `zfs mount -a` to do automounting, but I'll leave that for later.)

Now, properly speaking, the shepherd `file-systems` service should not be started until we have actually performed ZFS automounting.  This means that `file-systems` has to have as a requirement the ZFS shepherd service that implements the automounting.  And of course it should *not* require that if ZFS isn't installed in the system.

So, I made another patch that makes the `file-systems` shepherd service have an extensible set of requirements, like `user-processes` does.  It's below.  Actual `file-system`s declared in the operating system then make themselves requirements of `file-systems`.  Then, a ZFS automounting shepherd service can be installed by the `zfs-service-type` and added as a requirement of the `file-systems` shepherd service.

This is important since one possible use of ZFS is to have it provide the `/home` filesystem.  And the `/home` filesystem has to be mounted before `user-homes` shepherd service starts.  So the ZFS automounting shepherd service has to be a requirement of `file-systems`, which is enabled by the below.

Please review!


From 792a8f8efc95e4fe9a94d42f839ddcfb034b8540 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 08:15:54 +0800
Subject: [PATCH] gnu: Make file-systems target extensible by services.

* gnu/services/base.scm (file-system-shepherd-services): Move file-systems
shepherd service to ...
(file-systems-target-shepherd-services): ... new procedure here.
(file-systems-target-service-type): New variable.
(file-system-service-type): Extend file-systems-target service to add each
file-system as a requirement.
* gnu/system.scm (operating-system-default-essential-services): Instantiate
file-systems-target-service-type.
(hurd-default-essential-services): Instantiate file-systems-target-service-type.
---
 gnu/services/base.scm | 37 +++++++++++++++++++++++++++----------
 gnu/system.scm        |  2 ++
 2 files changed, 29 insertions(+), 10 deletions(-)

diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 945b546607..13cfb6a8a2 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -13,6 +13,7 @@
 ;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;; Copyright © 2020 Florian Pelz <pelzflorian@pelzflorian.de>
 ;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -67,6 +68,7 @@
   #:export (fstab-service-type
             root-file-system-service
             file-system-service-type
+            file-systems-target-service-type
             swap-service
             host-name-service
             console-keymap-service
@@ -362,18 +364,29 @@ FILE-SYSTEM."
                        (gnu system file-systems)
                        ,@%default-modules)))))))

+(define (file-systems-target-shepherd-services requirements)
+  (list
+    (shepherd-service
+      (provision '(file-systems))
+      (requirement (cons* 'root-file-system 'user-file-systems requirements))
+      (documentation "Target for all the initially-mounted file systems")
+      (start #~(const #t))
+      (stop #~(const #t)))))
+(define file-systems-target-service-type
+  (service-type
+    (name 'file-systems)
+    (extensions (list (service-extension shepherd-root-service-type
+                                         file-systems-target-shepherd-services)))
+    (compose concatenate)
+    (extend append)
+    ;; Extensions can add new values to this list.
+    (default-value '())
+    (description "The @code{file-systems} service is the target that is started
+when all file systems have been mounted.")))
+
 (define (file-system-shepherd-services file-systems)
   "Return the list of Shepherd services for FILE-SYSTEMS."
   (let* ((file-systems (filter file-system-mount? file-systems)))
-    (define sink
-      (shepherd-service
-       (provision '(file-systems))
-       (requirement (cons* 'root-file-system 'user-file-systems
-                           (map file-system->shepherd-service-name
-                                file-systems)))
-       (documentation "Target for all the initially-mounted file systems")
-       (start #~(const #t))
-       (stop #~(const #f))))

     (define known-mount-points
       (map file-system-mount-point file-systems))
@@ -403,7 +416,7 @@ FILE-SYSTEM."
                            (filter (negate known?) (mount-points)))
                  #f))))

-    (cons* sink user-unmount
+    (cons* user-unmount
            (map file-system-shepherd-service file-systems))))

 (define (file-system-fstab-entries file-systems)
@@ -431,6 +444,10 @@ FILE-SYSTEM."
                        (service-extension fstab-service-type
                                           file-system-fstab-entries)

+                       ;; Have 'file-systems' depend on each declared file system.
+                       (service-extension file-systems-target-service-type
+                                          (cut map file-system->shepherd-service-name <>))
+
                        ;; Have 'user-processes' depend on 'file-systems'.
                        (service-extension user-processes-service-type
                                           (const '(file-systems)))))
diff --git a/gnu/system.scm b/gnu/system.scm
index 5c530f176e..6987641ee8 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -667,6 +667,7 @@ bookkeeping."
                     (operating-system-setuid-programs os))
            (service profile-service-type
                     (operating-system-packages os))
+           (service file-systems-target-service-type)
            other-fs
            (append mappings swaps

@@ -691,6 +692,7 @@ bookkeeping."
                                    (operating-system-groups os))
                            (operating-system-skeletons os))
           (root-file-system-service)
+          (service file-systems-target-service-type)
           (service file-system-service-type '())
           (service fstab-service-type
                    (filter file-system-needed-for-boot?
--
2.29.2



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-06  5:20                     ` raid5atemyhomework
@ 2021-01-06 15:58                       ` raid5atemyhomework
  2021-01-09 18:14                         ` raid5atemyhomework
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-06 15:58 UTC (permalink / raw)
  To: guix-devel

Latest patchset: https://issues.guix.gnu.org/45692


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-06 15:58                       ` raid5atemyhomework
@ 2021-01-09 18:14                         ` raid5atemyhomework
  2021-01-10  5:17                           ` raid5atemyhomework
  0 siblings, 1 reply; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-09 18:14 UTC (permalink / raw)
  To: guix-devel

Hi guix-developers,

I just found out that ZFS on Linux maintains a file `/etc/zfs/zpool.cache` which contains information on ZPOOLs (i.e. ZFS-managed RAID arrays).  I just didn't encounter this file before on Guix because the file is created if and only if the directory `/etc/zfs/` exists, and that directory is not created by the ZFS installation process on Guix (because I'm the one making the ZFS installation process on Guix right now and I didn't know about this directory).

Now, my understanding is that `/etc/` directory is recreated at each `guix system reconfigure`.  Thus, if ZFS maintains information in `/etc/zfs/` then on a reconfigure the information is lost.

If so --- for Guix, what should I use instead?

As it happens, ZFS uses this directory for at least these things:

* `/etc/zfs/zpool.cache` - a binary file containing information about what ZPOOLs were created on this system.  The `zpool` command updates this file!
* `/etc/zfs/zpool.d/` - a directory of scripts that can be used to extend the `zpool` command; the ZFS release normally installs a bunch of files it has in those directories.
* `/etc/zfs/vdev_id.conf` - a sysad-managed configuration file that is used to indicate the paths.
* `/etc/zfs/zed.d/` - a directory of scripts that are executed when particular events are detected by the ZFS Event Daemon (which I didn't know about, and which should also be installed and started as a Shepherd service as well).  Sysads are supposed to link or copy files from `/libexec/zfs/zed.d/` (which are provided by the installation) to enable/disable particular zedlets, and can add other events here. The ZFS release normally links some of the files in `/libexec/zfs/zed.d/`.
* `/etc/zfs/zfs-functions` - a shell file containing functions that some bits of ZFS `init` scripts / `initramfs` scripts use, and maybe more. I think Guix can safely ignore this, though somebody (most likely me) will have to read it through to understand how ZFS actually implements some of its magical abilities.

Of these, many are stuff that we might plausibly generate by adding more fields in `zfs-configuration`, though the ZED and its zedlet system would bring in the possibility of writing Guile GEXP to create scripts for particular events.

However, I'm not so sure about `zpool.cache` file.  For one, it doesn't contain text data so it's not easy to generate it ourselves.

On the other hand I've got ZFS working without it, by just using `zpool import -a` to have ZFS scan for all devices.  The problem with this technique is if Guix is used on a system with several dozen or hundred devices in various ZPOOLs; this would slow down boot while ZFS is checking all arrays and figuring out which device goes into which array, whereas a `zpool.cache` file would let ZFS know about all pools created or imported on the system and would not need to scan their labels and reassemble their arrays and so on.  Another is that for a complex enough setup, a storage device might be connected to the hardware by hostile parties (for example, by network-addressable block devices, or USB) that contains a ZPOOL with a `mountpoint=/gnu/store` (or other sensitive directory) property, which, if auto-imported via `zpool import -a`, could let particular subdirectories of the filesystem to be subverted.

How best do I handle this?


Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: ZFS on Guix
  2021-01-09 18:14                         ` raid5atemyhomework
@ 2021-01-10  5:17                           ` raid5atemyhomework
  0 siblings, 0 replies; 15+ messages in thread
From: raid5atemyhomework @ 2021-01-10  5:17 UTC (permalink / raw)
  To: guix-devel

Hi guix-developers,

> Now, my understanding is that `/etc/` directory is recreated at each `guix system reconfigure`. Thus, if ZFS maintains information in `/etc/zfs/` then on a reconfigure the information is lost.
>
> If so --- for Guix, what should I use instead?
>


Okay, a `guix system reconfigure` doesn't remove files in `/etc`, at best I **think** it just copies/overwrites data, but if the `etc` directory of the built system doesn't have a file, it will not be overwritten if something else writes to `/etc`.  Is my understanding correct?

On the other hand --- is it "properly Guix" for a system component to write to a file in the `/etc` directory? Where should I put this kind of not-quite-configuration cache file?



Thanks
raid5atemyhomework


^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2021-01-10  5:18 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-02  6:16 ZFS on Guix raid5atemyhomework
2021-01-02  6:40 ` raid5atemyhomework
2021-01-03 15:50   ` Danny Milosavljevic
     [not found]     ` <Kyzl4xnRrDtWbTkAmBf1i8mtSZvu-AYatdazY1NsABFAzqqi7HQl-t0d2LWAInr8n7KxGyJWfIfTWqrefrxdgWelKbr2SWc9ATV5P8zrVtw=3D@protonmail.com>
     [not found]       ` <DmzbKm-kVn1tCfYc4eAXeEEbj3RN28qvXJaLL6dHtBBeNdXE-e-vN-a4Y-De38H8jI5lt19mauUTz9k0JArMyOT2ciYoxKY8mXFw3xeHGqo=3D@protonmail.com>
     [not found]         ` <guGWV=5FuA7tRqi0SzCiial7b5W7uKlmcMRhQXIlhtzyf1Fgq91eakr8d5EQrZ2fJPJK1OqKMCJfMgE2H4SjRSph-31ms=5F2g-mFiddv9ppiaQ=3D@protonmail.com>
2021-01-04  0:50     ` raid5atemyhomework
2021-01-04  1:15       ` raid5atemyhomework
2021-01-05 11:02         ` raid5atemyhomework
2021-01-05 14:56           ` raid5atemyhomework
2021-01-06  0:59             ` Carlo Zancanaro
2021-01-06  3:50               ` raid5atemyhomework
2021-01-06  3:58                 ` Carlo Zancanaro
     [not found]                   ` <t=5Ft9LHglTTJw2Bhtd2xX4JCJeZBi5Drqg0t04vJsfRLfoENqpftZcuHO8LYi2AO05P71lbbCgQC5etCDiPsdcssJmw1pHGyCY3 gUT-9w9?= =?us-ascii?Q?=5Fo=3D@protonmail.com>
     [not found]                     ` <Zd8uMcxWfNY1RxDn4gwrCZjHKAUxSQgcuBsNDAqa0tMqj=5FufQWE-URr7L49OBWMGDfQB8v=5F8eRdnhlsZTxU0Xq=5FF6tu-92jvvc1lSh-2tdA=3D@protonmail.com>
     [not found]                       ` <vzCMYS=5FrSzkd3ZDA5TktzybU2LmfZsjWLmrd0ABQ1bIKyulAreAghoDBo0yjb-bEbH5ZmKhOO3D9WPjuDoMMUs0O eUWA1WakV?= =?us-ascii?Q?WFo6H61IHY=3D@protonmail.com>
2021-01-06  4:41                   ` raid5atemyhomework
2021-01-06  5:20                     ` raid5atemyhomework
2021-01-06 15:58                       ` raid5atemyhomework
2021-01-09 18:14                         ` raid5atemyhomework
2021-01-10  5:17                           ` raid5atemyhomework
     [not found] <=5F1CLe9QSGsoMlu5WxBMXm4CbFLM=5FM9iRG1XQF9GDsK0GP208jpngdymfix4tAfoLP94mhMTt-Tx6OP2xN=5Fn78Jhx5KQzkiqPpIci=5F44C9OI=3D@protonmail.com>

unofficial mirror of guix-devel@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-devel/0 guix-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-devel guix-devel/ https://yhetil.org/guix-devel \
		guix-devel@gnu.org
	public-inbox-index guix-devel

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.devel
	nntp://news.gmane.io/gmane.comp.gnu.guix.devel


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git