unofficial mirror of guix-patches@gnu.org 
 help / color / mirror / Atom feed
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
@ 2021-01-06 15:52 raid5atemyhomework via Guix-patches via
  2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
                   ` (3 more replies)
  0 siblings, 4 replies; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:52 UTC (permalink / raw)
  To: 45692

Hi all,

Sorry for the excessive number of patches.

This patchset contains 4 patches, two that add new features to Guix and do not involve ZFS at all, one bugfix of the ZFS package, and one which makes installing ZFS on Guix easy.

* Patch 1: Create an extensible service type that installs kernel-loadable modules into the kernel.
  * This allows a ZFS service to add the ZFS module to the kernel.
* Patch 2: Make the `file-systems` shepherd service an extensible target, so that other services can add more requirements for `file-systems`.
  * This allows a ZFS service to make `file-systems` wait for the ZFS automount at startup, which is necessary in any case in order to ensure that ZFS has scanned for ZFS pools installed in the hardware.
* Patch 3: Fix the ZFS package definition.
* Patch 4: Add a ZFS service type that installs everything needed for basic ZFS usage!

I've tested the ZFS functionality in about a half-dozen VMs already, including checking that the ZFS setup persists across reboots of the VM:

* No ZFS installed.
* With ZFS installed but without any ZFS pools.
* With ZFS pool that is automounted.
* With ZFS pool in legacy mode (needs manual `mount -t zfs poolname /mountpoint`).
* Encrypted ZFS dataset (prompts passphrase at boot to mount).
* Encrypted ZFS dataseet as `/home` (prompts passphrase at boot to mount, user `/home` directories are created and maintained).

Other bits of ZFS functionality would be *nice* but aren't in this patchset yet (and at least for my purposes are not really necessary, except perhaps the auto-scrubbing, I just need to wrap my head around `mcron`...):

* Auto-scrubbing of ZFS pools to ensure no data corruption.
* Auto-snapshotting of ZFS datasets to protect against user error.
* `/` on ZFS.
* `/boot` on ZFS.
* Managing ZFS "legacy" mountpoints on  `(file-system ...)` declarations.
* Managing ZFS pool setup in the `operating-system` declaration somehow (not sure how to do this).

Thanks
raid5atemyhomework




^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules.
  2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
@ 2021-01-06 15:54 ` raid5atemyhomework via Guix-patches via
  2021-01-08 16:16   ` raid5atemyhomework via Guix-patches via
  2021-01-06 15:55 ` [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services raid5atemyhomework via Guix-patches via
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:54 UTC (permalink / raw)
  To: 45692

From 4beb73c62995cf236b402dad8e1c36016027c781 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH 1/4] gnu: Allow services to install kernel-loadable modules.

* gnu/system.scm (operating-system-directory-base-entries): Remove code
to handle generation of "kernel" and "hurd".
(operating-system-default-essential-services): Instantiate
kernel-loadable-module-service.
(hurd-default-essential-services): Instantiate
kernel-loadable-module-service.
(package-for-kernel): Move ...
* gnu/services.scm: ... to here.
(kernel-loadable-module-service-type): New variable.
(kernel-loadable-module-service): New procedure.
* gnu/tests/linux-modules.scm (run-loadable-kernel-modules-test): Move
code to ...
(run-loadable-kernel-modules-test-base): ... new procedure here.
(run-loadable-kernel-modules-service-test): New procedure.
(%test-loadable-kernel-modules-service-0): New variable.
(%test-loadable-kernel-modules-service-1): New variable.
(%test-loadable-kernel-modules-service-2): New variable.
* doc/guix.texi: Document kernel-loadable-module-service-type.
---
 doc/guix.texi               |  6 +++
 gnu/services.scm            | 70 ++++++++++++++++++++++++++++++++
 gnu/system.scm              | 37 +++++------------
 gnu/tests/linux-modules.scm | 81 ++++++++++++++++++++++++++++++++-----
 4 files changed, 157 insertions(+), 37 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 0f6e95a65a..78770151e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32409,6 +32409,12 @@ configuration when you use @command{guix system reconfigure},
 @command{guix system init}, or @command{guix deploy}.
 @end defvr

+@defvr {Scheme Variable} kernel-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+@end defvr
+
 @node Shepherd Services
 @subsection Shepherd Services

diff --git a/gnu/services.scm b/gnu/services.scm
index 13259dfaee..d7332a46b2 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2016 Chris Marusich <cmmarusich@gmail.com>
 ;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -33,6 +34,8 @@
   #:use-module (guix diagnostics)
   #:autoload   (guix openpgp) (openpgp-format-fingerprint)
   #:use-module (guix modules)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
   #:use-module (gnu packages base)
   #:use-module (gnu packages bash)
   #:use-module (gnu packages hurd)
@@ -75,6 +78,7 @@
             service-back-edges
             instantiate-missing-services
             fold-services
+            kernel-loadable-module-service

             service-error?
             missing-value-service-error?
@@ -106,6 +110,7 @@
             profile-service-type
             firmware-service-type
             gc-root-service-type
+            kernel-loadable-module-service-type

             %boot-service
             %activation-service
@@ -864,6 +869,71 @@ as Wifi cards.")))
 will not be reclaimed by the garbage collector.")
                 (default-value '())))

+;; Configuration for the kernel builder.
+(define-record-type* <kernel-builder-configuration> kernel-builder-configuration
+  make-kernel-builder-configuration
+  kernel-builder-configuration?
+  this-kernel-builder-configuration
+
+  (kernel   kernel-builder-configuration-kernel   (default #f))
+  (hurd     kernel-builder-configuration-hurd     (default #f))
+  (modules  kernel-builder-configuration-modules  (default '())))
+
+(define (package-for-kernel target-kernel module-package)
+  "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
+possible (that is if there's a LINUX keyword argument in the build system)."
+  (package
+    (inherit module-package)
+    (arguments
+     (substitute-keyword-arguments (package-arguments module-package)
+       ((#:linux kernel #f)
+        target-kernel)))))
+
+(define (kernel-builder-configuration->system-entry config)
+  "Return the kernel and hurd entries of the 'system' directory."
+  (mbegin %store-monad
+    (let* ((kernel  (kernel-builder-configuration-kernel config))
+           (hurd    (kernel-builder-configuration-hurd config))
+           (modules (kernel-builder-configuration-modules config))
+           (kernel  (if hurd
+                        kernel
+                        (profile
+                         (content (packages->manifest
+                                   (cons kernel
+                                         (map (lambda (module)
+                                                (if (package? module)
+                                                    (package-for-kernel kernel module)
+                                                    module))
+                                              modules))))
+                         (hooks (list linux-module-database))))))
+      (return `(("kernel" ,kernel)
+                ,@(if hurd `(("hurd" ,hurd)) '()))))))
+
+(define (kernel-builder-configuration-add-modules config modules)
+  "Constructs a kernel builder configuration that has its modules extended."
+  (kernel-builder-configuration
+    (inherit config)
+    (modules (append (kernel-builder-configuration-modules config) modules))))
+
+(define kernel-loadable-module-service-type
+  (service-type (name 'kernel-loadable-modules)
+                (extensions
+                 (list (service-extension system-service-type
+                                          kernel-builder-configuration->system-entry)))
+                (compose concatenate)
+                (extend kernel-builder-configuration-add-modules)
+                (description
+                 "Register packages containing kernel-loadable modules and adds them
+to the system.")))
+
+(define (kernel-loadable-module-service kernel hurd modules)
+  "Constructs the service that sets up kernel loadable modules."
+  (service kernel-loadable-module-service-type
+    (kernel-builder-configuration
+      (kernel kernel)
+      (hurd hurd)
+      (modules modules))))
+
 \f
 ;;;
 ;;; Service folding.
diff --git a/gnu/system.scm b/gnu/system.scm
index c284a18379..5c530f176e 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -12,6 +12,7 @@
 ;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <jannek@gnu.org>
 ;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -600,16 +601,6 @@ OS."
       (file-append (operating-system-kernel os)
                       "/" (system-linux-image-file-name))))

-(define (package-for-kernel target-kernel module-package)
-  "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
-possible (that is if there's a LINUX keyword argument in the build system)."
-  (package
-    (inherit module-package)
-    (arguments
-     (substitute-keyword-arguments (package-arguments module-package)
-       ((#:linux kernel #f)
-        target-kernel)))))
-
 (define %default-modprobe-blacklist
   ;; List of kernel modules to blacklist by default.
   '("usbmouse" ;races with bcm5974, see <https://bugs.gnu.org/35574>
@@ -625,26 +616,10 @@ possible (that is if there's a LINUX keyword argument in the build system)."
   "Return the basic entries of the 'system' directory of OS for use as the
 value of the SYSTEM-SERVICE-TYPE service."
   (let* ((locale  (operating-system-locale-directory os))
-         (kernel  (operating-system-kernel os))
          (hurd    (operating-system-hurd os))
-         (modules (operating-system-kernel-loadable-modules os))
-         (kernel  (if hurd
-                      kernel
-                      (profile
-                       (content (packages->manifest
-                                 (cons kernel
-                                       (map (lambda (module)
-                                              (if (package? module)
-                                                  (package-for-kernel kernel
-                                                                      module)
-                                                  module))
-                                            modules))))
-                       (hooks (list linux-module-database)))))
          (initrd  (and (not hurd) (operating-system-initrd-file os)))
          (params  (operating-system-boot-parameters-file os)))
-    `(("kernel" ,kernel)
-      ,@(if hurd `(("hurd" ,hurd)) '())
-      ("parameters" ,params)
+    `(("parameters" ,params)
       ,@(if initrd `(("initrd" ,initrd)) '())
       ("locale" ,locale))))   ;used by libc

@@ -663,6 +638,10 @@ bookkeeping."
          (host-name (host-name-service (operating-system-host-name os)))
          (entries   (operating-system-directory-base-entries os)))
     (cons* (service system-service-type entries)
+           (kernel-loadable-module-service
+             (operating-system-kernel os)
+             (operating-system-hurd os)
+             (operating-system-kernel-loadable-modules os))
            %boot-service

            ;; %SHEPHERD-ROOT-SERVICE must come last so that the gexp that
@@ -699,6 +678,10 @@ bookkeeping."
 (define (hurd-default-essential-services os)
   (let ((entries (operating-system-directory-base-entries os)))
     (list (service system-service-type entries)
+          (kernel-loadable-module-service
+            (operating-system-kernel os)
+            (operating-system-hurd os)
+            (operating-system-kernel-loadable-modules os))
           %boot-service
           %hurd-startup-service
           %activation-service
diff --git a/gnu/tests/linux-modules.scm b/gnu/tests/linux-modules.scm
index 953b132ef7..9739e4124d 100644
--- a/gnu/tests/linux-modules.scm
+++ b/gnu/tests/linux-modules.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2019 Jakob L. Kreuze <zerodaysfordays@sdf.org>
 ;;; Copyright © 2020 Danny Milosavljevic <dannym@scratchpost.org>
 ;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -34,7 +35,10 @@
   #:use-module (guix utils)
   #:export (%test-loadable-kernel-modules-0
             %test-loadable-kernel-modules-1
-            %test-loadable-kernel-modules-2))
+            %test-loadable-kernel-modules-2
+            %test-loadable-kernel-modules-service-0
+            %test-loadable-kernel-modules-service-1
+            %test-loadable-kernel-modules-service-2))

 ;;; Commentary:
 ;;;
@@ -66,17 +70,11 @@ that MODULES are actually loaded."
                        (member module modules string=?))
                      '#$modules))))))

-(define* (run-loadable-kernel-modules-test module-packages module-names)
-  "Run a test of an OS having MODULE-PACKAGES, and verify that MODULE-NAMES
-are loaded in memory."
+(define* (run-loadable-kernel-modules-test-base base-os module-names)
+  "Run a test of BASE-OS, verifying that MODULE-NAMES are loaded in memory."
   (define os
     (marionette-operating-system
-     (operating-system
-      (inherit (simple-operating-system))
-      (services (cons (service kernel-module-loader-service-type module-names)
-                      (operating-system-user-services
-                       (simple-operating-system))))
-      (kernel-loadable-modules module-packages))
+     base-os
      #:imported-modules '((guix combinators))))
   (define vm (virtual-machine os))
   (define (test script)
@@ -98,6 +96,37 @@ are loaded in memory."
   (gexp->derivation "loadable-kernel-modules"
                     (test (modules-loaded?-program os module-names))))

+(define* (run-loadable-kernel-modules-test module-packages module-names)
+  "Run a test of an OS having MODULE-PACKAGES, and verify that MODULE-NAMES
+are loaded in memory."
+  (run-loadable-kernel-modules-test-base
+    (operating-system
+      (inherit (simple-operating-system))
+      (services (cons (service kernel-module-loader-service-type module-names)
+                      (operating-system-user-services
+                       (simple-operating-system))))
+      (kernel-loadable-modules module-packages))
+    module-names))
+
+(define* (run-loadable-kernel-modules-service-test module-packages module-names)
+  "Run a test of an OS having MODULE-PACKAGES, which are loaded by creating a
+service that extends KERNEL-LOADABLE-MODULE-SERVICE-TYPE. Then verify that
+MODULE-NAMES are loaded in memory."
+  (define module-installing-service-type
+    (service-type
+      (name 'module-installing-service)
+      (extensions (list (service-extension kernel-loadable-module-service-type
+                                           (const module-packages))))
+      (default-value #f)))
+  (run-loadable-kernel-modules-test-base
+    (operating-system
+      (inherit (simple-operating-system))
+      (services (cons* (service kernel-module-loader-service-type module-names)
+                       (service module-installing-service-type)
+                       (operating-system-user-services
+                        (simple-operating-system)))))
+    module-names))
+
 (define %test-loadable-kernel-modules-0
   (system-test
    (name "loadable-kernel-modules-0")
@@ -129,3 +158,35 @@ with two extra modules.")
                                                  (package-arguments
                                                   ddcci-driver-linux))))))
            '("acpi_call" "ddcci")))))
+
+(define %test-loadable-kernel-modules-service-0
+  (system-test
+   (name "loadable-kernel-modules-service-0")
+   (description "Tests loadable kernel modules extensible service with no
+extra modules.")
+   (value (run-loadable-kernel-modules-service-test '() '()))))
+
+(define %test-loadable-kernel-modules-service-1
+  (system-test
+   (name "loadable-kernel-modules-service-1")
+   (description "Tests loadable kernel modules extensible service with one
+extra module.")
+   (value (run-loadable-kernel-modules-service-test
+           (list ddcci-driver-linux)
+           '("ddcci")))))
+
+(define %test-loadable-kernel-modules-service-2
+  (system-test
+   (name "loadable-kernel-modules-service-2")
+   (description "Tests loadable kernel modules extensible service with two
+extra modules.")
+   (value (run-loadable-kernel-modules-service-test
+           (list acpi-call-linux-module
+                 (package
+                   (inherit ddcci-driver-linux)
+                   (arguments
+                    `(#:linux #f
+                      ,@(strip-keyword-arguments '(#:linux)
+                                                 (package-arguments
+                                                  ddcci-driver-linux))))))
+           '("acpi_call" "ddcci")))))
--
2.29.2





^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services.
  2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
  2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
@ 2021-01-06 15:55 ` raid5atemyhomework via Guix-patches via
  2021-01-06 15:56 ` [bug#45692] [PATCH 3/4] gnu: Fix ZFS package raid5atemyhomework via Guix-patches via
  2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
  3 siblings, 0 replies; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:55 UTC (permalink / raw)
  To: 45692

From 792a8f8efc95e4fe9a94d42f839ddcfb034b8540 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 08:15:54 +0800
Subject: [PATCH 2/4] gnu: Make file-systems target extensible by services.

* gnu/services/base.scm (file-system-shepherd-services): Move file-systems
shepherd service to ...
(file-systems-target-shepherd-services): ... new procedure here.
(file-systems-target-service-type): New variable.
(file-system-service-type): Extend file-systems-target service to add each
file-system as a requirement.
* gnu/system.scm (operating-system-default-essential-services): Instantiate
file-systems-target-service-type.
(hurd-default-essential-services): Instantiate file-systems-target-service-type.
---
 gnu/services/base.scm | 37 +++++++++++++++++++++++++++----------
 gnu/system.scm        |  2 ++
 2 files changed, 29 insertions(+), 10 deletions(-)

diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 945b546607..13cfb6a8a2 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -13,6 +13,7 @@
 ;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;; Copyright © 2020 Florian Pelz <pelzflorian@pelzflorian.de>
 ;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -67,6 +68,7 @@
   #:export (fstab-service-type
             root-file-system-service
             file-system-service-type
+            file-systems-target-service-type
             swap-service
             host-name-service
             console-keymap-service
@@ -362,18 +364,29 @@ FILE-SYSTEM."
                        (gnu system file-systems)
                        ,@%default-modules)))))))

+(define (file-systems-target-shepherd-services requirements)
+  (list
+    (shepherd-service
+      (provision '(file-systems))
+      (requirement (cons* 'root-file-system 'user-file-systems requirements))
+      (documentation "Target for all the initially-mounted file systems")
+      (start #~(const #t))
+      (stop #~(const #t)))))
+(define file-systems-target-service-type
+  (service-type
+    (name 'file-systems)
+    (extensions (list (service-extension shepherd-root-service-type
+                                         file-systems-target-shepherd-services)))
+    (compose concatenate)
+    (extend append)
+    ;; Extensions can add new values to this list.
+    (default-value '())
+    (description "The @code{file-systems} service is the target that is started
+when all file systems have been mounted.")))
+
 (define (file-system-shepherd-services file-systems)
   "Return the list of Shepherd services for FILE-SYSTEMS."
   (let* ((file-systems (filter file-system-mount? file-systems)))
-    (define sink
-      (shepherd-service
-       (provision '(file-systems))
-       (requirement (cons* 'root-file-system 'user-file-systems
-                           (map file-system->shepherd-service-name
-                                file-systems)))
-       (documentation "Target for all the initially-mounted file systems")
-       (start #~(const #t))
-       (stop #~(const #f))))

     (define known-mount-points
       (map file-system-mount-point file-systems))
@@ -403,7 +416,7 @@ FILE-SYSTEM."
                            (filter (negate known?) (mount-points)))
                  #f))))

-    (cons* sink user-unmount
+    (cons* user-unmount
            (map file-system-shepherd-service file-systems))))

 (define (file-system-fstab-entries file-systems)
@@ -431,6 +444,10 @@ FILE-SYSTEM."
                        (service-extension fstab-service-type
                                           file-system-fstab-entries)

+                       ;; Have 'file-systems' depend on each declared file system.
+                       (service-extension file-systems-target-service-type
+                                          (cut map file-system->shepherd-service-name <>))
+
                        ;; Have 'user-processes' depend on 'file-systems'.
                        (service-extension user-processes-service-type
                                           (const '(file-systems)))))
diff --git a/gnu/system.scm b/gnu/system.scm
index 5c530f176e..6987641ee8 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -667,6 +667,7 @@ bookkeeping."
                     (operating-system-setuid-programs os))
            (service profile-service-type
                     (operating-system-packages os))
+           (service file-systems-target-service-type)
            other-fs
            (append mappings swaps

@@ -691,6 +692,7 @@ bookkeeping."
                                    (operating-system-groups os))
                            (operating-system-skeletons os))
           (root-file-system-service)
+          (service file-systems-target-service-type)
           (service file-system-service-type '())
           (service fstab-service-type
                    (filter file-system-needed-for-boot?
--
2.29.2





^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] [PATCH 3/4] gnu: Fix ZFS package.
  2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
  2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
  2021-01-06 15:55 ` [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services raid5atemyhomework via Guix-patches via
@ 2021-01-06 15:56 ` raid5atemyhomework via Guix-patches via
  2021-01-07  8:23   ` Danny Milosavljevic
  2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
  3 siblings, 1 reply; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:56 UTC (permalink / raw)
  To: 45692

From d1c88161f2a32f4c03de57ee9d7582cc6c40f6bc Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Tue, 5 Jan 2021 20:00:50 +0800
Subject: [PATCH 3/4] gnu: Fix ZFS package.

* gnu/packages/file-systems.scm (zfs): Correct "util-linux" input, add
separate "util-linux:lib" input.
---
 gnu/packages/file-systems.scm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 895ad069c5..2c5ad95d63 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -6,6 +6,7 @@
 ;;; Copyright © 2019, 2020 Efraim Flashner <efraim@flashner.co.il>
 ;;; Copyright © 2020 Raghav Gururajan <raghavgururajan@disroot.org>
 ;;; Copyright © 2020 Morgan Smith <Morgan.J.Smith@outlook.com>
+;;; Copyright © 2021 raid5atemyhoemwork <raid5atemyhomework@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -944,7 +945,8 @@ APFS.")
        ("openssl" ,openssl)
        ("python" ,python)
        ("python-cffi" ,python-cffi)
-       ("util-linux" ,util-linux "lib")
+       ("util-linux" ,util-linux)
+       ("util-linux:lib" ,util-linux "lib")
        ("zlib" ,zlib)))
     (home-page "https://zfsonlinux.org/")
     (synopsis "Native ZFS on Linux")
--
2.29.2





^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] [PATCH 4/4] gnu: Add ZFS service.
  2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
                   ` (2 preceding siblings ...)
  2021-01-06 15:56 ` [bug#45692] [PATCH 3/4] gnu: Fix ZFS package raid5atemyhomework via Guix-patches via
@ 2021-01-06 15:57 ` raid5atemyhomework via Guix-patches via
  2021-01-06 19:41   ` [bug#45703] kernel-module-configuration-service for configuring kernel parameters Danny Milosavljevic
  2021-01-08 15:02   ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
  3 siblings, 2 replies; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:57 UTC (permalink / raw)
  To: 45692

From deded39fe88c44a18b0b66bab8c4300aca4c387e Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 09:24:20 +0800
Subject: [PATCH 4/4] gnu: Add ZFS service.

* gnu/services/file-systems.scm: New file.
(zfs-service-type): New variable.
(zfs-configuration): New variable.
(zfs-configuration?): New procedure.
* gnu/local.mk: Add gnu/services/file-systems.scm.
* doc/guix.texi (ZFS file system): New subsection.
---
 doc/guix.texi                 | 107 ++++++++++++++++++++++++++
 gnu/local.mk                  |   2 +
 gnu/services/file-systems.scm | 136 ++++++++++++++++++++++++++++++++++
 3 files changed, 245 insertions(+)
 create mode 100644 gnu/services/file-systems.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index 78770151e3..62437083cb 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13885,6 +13885,113 @@ a file system declaration such as:
 compress-force=zstd,space_cache=v2"))
 @end lisp

+@node ZFS file system
+@subsection ZFS file system
+
+The ZFS file system has a license incompatible with the Linux kernel,
+and thus cannot be distributed with the kernel. However, as a user
+you have the right to do whatever you want on your own hardware,
+including download the ZFS source code, compile it, link it to your
+own private copy of Linux, and run it. You can even use the Guix build
+system to automate this.
+
+As a large and complex kernel module, ZFS on Linux has to be compiled
+with a specific version of the kernel. Often even the latest ZFS
+package available cannot be compiled with the latest Linux kernel
+package provided by Guix System. Thus, installing the @code{zfs}
+package is likely to fail.
+
+Instead, you have to use an older long-term-support Linux kernel.
+Do not use @code{linux-libre-lts} as the latest long-term-support
+kernel might be too new for the ZFS package; instead, explicitly
+select the version number, like @code{linux-libre-5.4}.
+
+Then, you have to modify your system configuration file and use the
+selected older kernel, and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+   ; @dots{}
+   linux)
+(use-service-modules
+   ; @dots{}
+   file-systems)
+
+(define my-kernel linux-libre-5.4)
+
+(operating-system
+  (kernel my-kernel)
+  ;; @dots{}
+  (services
+    (cons* (service zfs-service-type
+             (zfs-configuration
+               (kernel my-kernel)))
+           ; @dots{}
+           %desktop-services))
+  ;; @dots{}
+  )
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type of the service to compile and install OpenZFS to
+your operating system. It loads the ZFS module at startup, imports
+pools, mounts automounted ZFS datasets, installs the ZFS command
+line tools, and can provide module options for the ZFS module.  Its
+value must be a @code{zfs-configuration} record (see below).
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+  (zfs-configuration
+    (kernel linux-libre-5.4)
+    (options '("zfs_arc_max=5000000000"))))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS service.
+The available fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux kernel to compile ZFS for. Required.
+
+@item @code{base-zfs} (default: @code{zfs})
+The ZFS package to use. It will be modified to use the indicated
+kernel.
+
+@item @code{options} (default: @code{'()})
+A list of string options to pass as options to the ZFS module.
+These will be put in a @file{/etc/modprobe.d/zfs.conf} file,
+for example setting this to @code{'("zfs_admin_snapshot=1"
+"zfs_trim_extent_bytes_min=0")} will create the following file:
+
+@example
+options zfs zfs_admin_snapshot=1 zfs_trim_extent_bytes_min=0
+@end example
+@end table
+@end deftp
+
+Once your system has been configured to include the ZFS service
+and you have restarted the system, you can manage ZFS pools and
+datasets with @code{zpool} and @code{zfs} commands.
+
+ZFS datasets with an appropriate @code{mountpoint} property will
+be automounted at startup after the root file system is started.
+Encrypted datasets that are automounted will cause boot to pause
+and prompt for the password to be provided on the console.
+
+It's possible to have a ZFS dataset as your @code{/home} by simply
+setting the @code{mountpoint} property. However, note that ZFS will
+refuse to mount over a non-empty directory, so if your root
+filesystem already has a non-empty @code{/home}, you should remove
+it and its contents, then restart the system.
+
+Having ZFS as a root filesystem or as @code{/boot} is not supported
+yet.
+
 @node Mapped Devices
 @section Mapped Devices

diff --git a/gnu/local.mk b/gnu/local.mk
index 53a67e6ffe..49073ac2ac 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -39,6 +39,7 @@
 # Copyright © 2020 Martin Becze <mjbecze@riseup.net>
 # Copyright © 2020 Malte Frank Gerdes <mate.f.gerdes@gmail.com>
 # Copyright © 2020 Vinicius Monego <monego@posteo.net>
+# Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
 #
 # This file is part of GNU Guix.
 #
@@ -601,6 +602,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/services/dict.scm				\
   %D%/services/dns.scm				\
   %D%/services/docker.scm			\
+  %D%/services/file-systems.scm			\
   %D%/services/authentication.scm		\
   %D%/services/games.scm			\
   %D%/services/ganeti.scm			\
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..7a92a2b9d5
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,136 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+  #:use-module (gnu packages file-systems)
+  #:use-module (gnu services)
+  #:use-module (gnu services base)
+  #:use-module (gnu services shepherd)
+  #:use-module (guix gexp)
+  #:use-module (guix packages)
+  #:use-module (guix records)
+  #:export (zfs-service-type
+            zfs-configuration
+            zfs-configuration?))
+
+(define-record-type* <zfs-configuration>
+  zfs-configuration make-zfs-configuration zfs-configuration?
+  ; kernel you want to compile the base-zfs module for.
+  (kernel     zfs-configuration-kernel)
+  ; base package that will be compiled for the kernel
+  (base-zfs   zfs-configuration-base-zfs  (default zfs))
+  ; list of string options.
+  (options    zfs-configuration-options   (default '())))
+
+(define (make-zfs-package conf)
+  (let ((base-zfs (zfs-configuration-base-zfs conf))
+        (kernel   (zfs-configuration-kernel conf)))
+    (package
+      (inherit base-zfs)
+      (name (string-join (list (package-name base-zfs)
+                               "for"
+                               (package-name kernel)
+                               (package-version kernel)
+                               "version")
+                         "-"))
+      (arguments (cons* #:linux kernel (package-arguments base-zfs))))))
+
+(define (zfs-loadable-module conf)
+  (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+  (let* ((zfs-package (make-zfs-package conf))
+         (zpool       (file-append zfs-package "/sbin/zpool")))
+    (list
+      (shepherd-service
+        (documentation "Scans for ZFS pools and automounts filesystems.")
+        (provision '(zfs-scan-automount))
+        (requirement '(root-file-system))
+        (modules `((srfi srfi-1)
+                   (srfi srfi-34)
+                   (srfi srfi-35)
+                   (rnrs io ports)
+                   ,@%default-modules))
+        (start #~(lambda _
+                   (and
+                     ;; You'd think we could've used kernel-module-loader-service-type,
+                     ;; but the kernel-module-loader shepherd service is dependent on
+                     ;; file-systems, and file-systems is made dependent on this
+                     ;; service. And we need the kernel module to be loaded before we
+                     ;; scan for ZFS pools. So break the dependency loop by just
+                     ;; loading ZFS module here by ourselves.
+                     (or (file-exists? "/proc/sys/kernel/modprobe")
+                         (begin
+                           (format (current-error-port) "error loading 'zfs' module: ~a~%"
+                                   "Kernel is missing loadable module support.")
+                           #f))
+                     (guard (c ((message-condition? c)
+                                (format (current-error-port)
+                                        "error loading 'zfs' module: ~a~%"
+                                        (condition-message c))
+                                #f))
+                       (let ((modprobe (call-with-input-file "/proc/sys/kernel/modprobe"
+                                                             get-line)))
+                         (invoke/quiet modprobe "--" "zfs")))
+
+                     ; scan for pools and automount contained datasets.
+                     (guard (c ((message-condition? c)
+                                (format (current-error-port)
+                                        "error importing zpools: ~a~%"
+                                        (condition-message?))
+                                #f))
+                       ;; (current-output-port) is typically connected to /dev/klog,
+                       ;; so redirect it to (current-error-port) so that user can see
+                       ;; prompts for passphrases on console
+                       (with-output-to-port (current-error-port)
+                         (lambda ()
+                           (invoke #$zpool "import" "-a" "-l")))))))
+        (stop #~(const #t))))))
+
+(define (zfs-profile-service conf)
+  (list (make-zfs-package conf)))
+
+(define (zfs-etc-service conf)
+  (let ((options (zfs-configuration-options conf)))
+    (if (null? options)
+        '()
+        `(("modprobe.d/zfs.conf"
+           ,(plain-file "zfs.conf"
+                        (string-join (cons "options zfs" options) " ")))))))
+
+(define zfs-service-type
+  (service-type (name 'zfs)
+                (extensions
+                  (list
+                    ; install the kernel module
+                    (service-extension kernel-loadable-module-service-type
+                                       zfs-loadable-module)
+                    ; load ZFS module, scan ZFS pools, and automount filesystems
+                    (service-extension shepherd-root-service-type
+                                       zfs-shepherd-services)
+                    ; make sure automount occurs before file-systems target is reached
+                    (service-extension file-systems-target-service-type
+                                       (const '(zfs-scan-automount)))
+                    ; install ZFS management tools
+                    (service-extension profile-service-type
+                                       zfs-profile-service)
+                    ; install ZFS module options
+                    (service-extension etc-service-type
+                                       zfs-etc-service)))
+                (description
+                  "Install ZFS, an advanced filesystem and volume manager.")))
--
2.29.2





^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45703] kernel-module-configuration-service for configuring kernel parameters
  2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
@ 2021-01-06 19:41   ` Danny Milosavljevic
  2021-01-07  0:04     ` raid5atemyhomework via Guix-patches via
  2021-01-08 15:02   ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
  1 sibling, 1 reply; 13+ messages in thread
From: Danny Milosavljevic @ 2021-01-06 19:41 UTC (permalink / raw)
  Cc: raid5atemyhomework, 45703, 45692

[-- Attachment #1: Type: text/plain, Size: 5916 bytes --]

Hi raid5atemyhomework,
Hi everyone,

@raid5atemyhomework, thanks for all the patches!

On Wed, 06 Jan 2021 15:57:19 +0000
raid5atemyhomework via Guix-patches via <guix-patches@gnu.org> wrote:

> +@item @code{options} (default: @code{'()})
> +A list of string options to pass as options to the ZFS module.

> +These will be put in a @file{/etc/modprobe.d/zfs.conf} file,
> +for example setting this to @code{'("zfs_admin_snapshot=1"
> +"zfs_trim_extent_bytes_min=0")} will create the following file:

Sure, but it would be better to create a way to configure those module parameters
in Guix in a declarative way, first.

Your new kernel-loadable-module-service-type would be a good template
to write a kernel-module-configuration-service that can be extended by
other services.  The latter should allow users to parametrize the kernel
in general via the operating-system form, and also allow services to extend
such configuration (with merging, and conflict detection).

It would be used:

(1) To declaratively specify the contents of something like /etc/modprobe.d .
It shouldn't even be called "/etc/modprobe.d"--it should also be in the store
instead.  This directory is only useful for non-Linux-builtins.

(2) If the "module" is built-in then the kernel command line must get the
options instead.  (in fact, it works as a kernel command line option also
if it's a loadable module--so not sure we need /etc/modprobe.d at all--at
least at first.  But there's probably a maximal length for the kernel command
line that we could exceed if we did that long term)

I know it's annoying that Guix doesn't have this facility already, but the
time to introduce an interface for /etc/modprobe.d and the kernel command
line for builtin modules is before other services introduce their own
ad hoc way to create /etc/modprobe.d--like this tries to do here.

See also https://issues.guix.info/issue/42193 for an earlier attempt (which
is already very far--but it has a bug somewhere).  There's also already a
kernel profile thing like you wrote in that patchset.
(Note that I would prefer there not to be a "LOAD?" in there because it
confuses loading the module (which is usually NOT started by user space
but by the kernel on its own) and confguring the module (which has to be
done by user space because it's specifying policy, not mechanism))

Also, because the kernel usually loads loadable modules on its own (potentially
really early), /etc/modprobe.d has to be preset and known to the modprobe
executable VERY EARLY (via environment variable MODPROBE_OPTIONS--see 
gnu/services.scm %modprobe-wrapper).

It is totally possible that some modules in the initrd need options, too
(see load-linux-modules-from-directory for where this would need to go).
load-linux-module/fd already accepts options and flags--but both are not given
on the call.  For this, part of future kernel-module-configuration entries
(the ones needed for modules in the initrd) should be copied into the initrd,
too.

Then there's the handoff between initrd and main system.  It would be bad
if the kernel tried and succeeded to load a module that is not in the initrd
just before the modprobe.d directory is set up (because it would be loaded
without passing the options the user configured)--so that needs to be avoided.

@ludo: Could you help here?

> +@example
> +options zfs zfs_admin_snapshot=1 zfs_trim_extent_bytes_min=0

Note:

This can be usefully put in a modprobe.d-like directory if zfs is a module, but
not if it's built into the kernel.  But it can be put into the kernel command
line in both cases.

But I guess the ZFS Linux kernel module can't be built-in into the kernel
anyway.

But that's a special case--in general, it's very much possible to make modules
built-in.

> +                     ;; You'd think we could've used kernel-module-loader-service-type,

Definitely.

> +                     ;; but the kernel-module-loader shepherd service is dependent on
> +                     ;; file-systems, 

Yes--but why is that dependent on 'file-systems ? Is it because it needs /proc ?
Or is it an oversight ?  I would prefer to get rid of this dependency and then
use kernel-module-loader-service-type.

Also, this manual loading of kernel modules is not supposed to be the way to
do things in Linux.  That a kernel module was compiled as a module is
an *implementation detail*--so Linux should (and usually does) automatically
load kernel modules the first time a device for them is accessed (after all,
how would user space know whether something is compiled as a module or
built-in--that would be too much to ask).

Linux is not a microkernel, so the kind of modularily modprobe.d suggests
exists does not in fact exist in kernel space--even though Linux does a good
job faking it:  modprobe.d contains:

* "alias": a feature to configure aliases, with wildcards (only one level of
aliases allowed!)
* "options" per module (also works for aliases with wildcards!  That will be
"fun" to map to Guix)
* "install" in order to run some custom executable instead of loading the
module.
* "remove" in order to run some custom executable instead of unloading the
module.
* "blacklist" to ignore specific internal aliases of a module (that does
not do what one would intuitively think!).

If the file name of the regular file under /etc/modprobe.d is not used for
anything, then we can just have one file /gnu/store/*modprobe.d/guix.conf
in total in there.

Then there are sysctl kernel parameters--but those Guix already exposes via
sysctl-service-type.  But those should also be made able to be extended
by other services, and merge conflicts should be handled.  For example, users
often set net.ipv4.ip_forward=1 (for example via sysctl).

Thank you for all your effort to make ZFS work nicely in Guix.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45703] kernel-module-configuration-service for configuring kernel parameters
  2021-01-06 19:41   ` [bug#45703] kernel-module-configuration-service for configuring kernel parameters Danny Milosavljevic
@ 2021-01-07  0:04     ` raid5atemyhomework via Guix-patches via
  2021-01-07  5:38       ` [bug#45692] " raid5atemyhomework via Guix-patches via
  0 siblings, 1 reply; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-07  0:04 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: 45703, 45692

Hello Danny,


> See also https://issues.guix.info/issue/42193 for an earlier attempt (which
> is already very far--but it has a bug somewhere). There's also already a
> kernel profile thing like you wrote in that patchset.
> (Note that I would prefer there not to be a "LOAD?" in there because it
> confuses loading the module (which is usually NOT started by user space
> but by the kernel on its own) and confguring the module (which has to be
> done by user space because it's specifying policy, not mechanism))

Looks like that patchset was merged in, so basically I can just depend on that? So the first patch in this patchset would be dropped?

> But I guess the ZFS Linux kernel module can't be built-in into the kernel
> anyway.
>
> But that's a special case--in general, it's very much possible to make modules
> built-in.

ZFS *can* be built-in to the kernel, Ubuntu does it. You can't distribute it like that (Ubuntu distributes it like that but presumably they have enough lawyers to muddy the waters so that they can get away with it), but as the documentation in this patch notes: the user has every right to do whatever they want on the machine they own, including build a Linux kernel that has ZFS built-in and run it, they just can't make that version available to somebody else.

So to go whole-hog, we would have a service that replaces the kernel package and inserts kernel module sources in-tree somehow, then compiles Linux-libre on the user's machine. That would probably be a lot more painful to install ZFS with (the user has to recompile the whole kernel at each update of either kernel or ZFS, whereas with a kernel module the user has to recompile just the kernel module), so maybe kernel module is still better overall.


> > -                       ;; You'd think we could've used kernel-module-loader-service-type,
> >
> >
>
> Definitely.
>
> > -                       ;; but the kernel-module-loader shepherd service is dependent on
> >
> >
> > -                       ;; file-systems,
> >
> >
>
> Yes--but why is that dependent on 'file-systems ? Is it because it needs /proc ?
> Or is it an oversight ? I would prefer to get rid of this dependency and then
> use kernel-module-loader-service-type.

Dunno --- one VM I tested, I removed the `zfs-scan-automount` shepherd service from the `file-systems` target, and the VM still wouldn't boot, claiming a stack overflow (the same error which I got when I was still trying to use kernel-module-loader-service-type here).  Or maybe I just got confused with which VM was which, testing VMs wasn't a stress-free vacation.  I just want ZFS, because MD RAID5 ate my homework, this is getting tiresome...

One thing I notice about `kernel-module-loader-service-type` is that it's not instantiated in essential services, or indeed anywhere in Guix.  A few services *do* extend it.  But my **very rough** understanding is that if you're going to extend a service, it had better be instantiated *once* in the list of services.

In particular I note that the documentation for `kernel-module-loader-service-type` shows an example where it uses `service` to program the `kernel-module-loader-service-type`, not `simple-service`.  This suggests to me that `kernel-module-loader-service-type` is broken because it's not in the list of essential services but is extensible.  Maybe.  It's designed as an extensible service, but isn't instantiated at default.  Maybe that's what really bit me and not the shepherd circular dependency loop? *shrug*


>
> Also, this manual loading of kernel modules is not supposed to be the way to
> do things in Linux. That a kernel module was compiled as a module is
> an implementation detail--so Linux should (and usually does) automatically
> load kernel modules the first time a device for them is accessed (after all,
> how would user space know whether something is compiled as a module or
> built-in--that would be too much to ask).

So how do I get ZFS loaded? Note that the devices it targets are block devices and it needs to scan for block devices that are formatted for ZFS.  Do other filesystems have some autoload rule?

Thanks
raid5atemyhomework




^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] kernel-module-configuration-service for configuring kernel parameters
  2021-01-07  0:04     ` raid5atemyhomework via Guix-patches via
@ 2021-01-07  5:38       ` raid5atemyhomework via Guix-patches via
  2021-01-07  9:16         ` [bug#42193] " raid5atemyhomework via Guix-patches via
  0 siblings, 1 reply; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-07  5:38 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: 45692

Hi Danny,

> > See also https://issues.guix.info/issue/42193 for an earlier attempt (which
> > is already very far--but it has a bug somewhere). There's also already a
> > kernel profile thing like you wrote in that patchset.
> > (Note that I would prefer there not to be a "LOAD?" in there because it
> > confuses loading the module (which is usually NOT started by user space
> > but by the kernel on its own) and confguring the module (which has to be
> > done by user space because it's specifying policy, not mechanism))
>
> Looks like that patchset was merged in, so basically I can just depend on that? So the first patch in this patchset would be dropped?

No, sorry, my mistake, only one patch completely unrelated to the actual new service type was merged in.

In any case --- is this objection something that would block this patchset from being added to Guix? I appreciate that we should "do it right" --- but there's also an argument for "keep it simple", and the first patch in this patchset gets us a good part of the way to what is needed.

It seems that my patch is equivalent to the existing WIP 2/6 `kernel-profile-service-type`, so maybe I can just steal that patch for now, write tests specific for it, and get on with running ZFS in production?

Can we at least get patch 3/4 of *this* patchset merged since it's a trivial bugfix?  How about patch 2/4, which is not so trivial, but does give some flexibility in case other filesystems want to be as ambitious as ZFS is?

I imagine a later, much more comprehensive `kernel-module-configuration-service-type` can be built on top of `kernel-profile-service-type`, `kernel-module-loader-service-type`, and a later `kernel-module-options-service-type`.  Then `zfs-service-type` can be modified to use that `kernel-module-configuration-service-type` instead of rolling its own bits and pieces.

----

On the other hand, if we want to think of `/` on ZFS, then we need a notion of kernel modules that are added to the `initrd` file, rather than to the `kernel` directory.

Diving into `gnu/system/linux-initrd.scm`, I note that we can provide a kernel package and a list of modules to copy from the kernel package to the `initrd` by a `(flat-linux-module-directory linux modules)` function.

I imagine that it would be possible to modify this as follows:

* Have `flat-linux-module-directory` accept a list of packages from which to find module names, not just a single package.
* Remove more code from `(operating-system-directory-base-entries os)`, and put the creation of the `"initrd"` file into a service, in much the same way that my patch does (unless there's an existing more-guixy way of putting files into `initrd`?).
  * Create a service type that gathers packages whose modules are to be added to the `initrd`, and if that list is non-empty, pass it to a new key `#:extra-linux-module-packages` of the `initrd` function.
    * `raw-initrd` would then pass that field together with the `linux` argument to the modified `flat-linux-module-directory` procedure.
  * Create another service type that gathers module names to be loaded at `boot` and appends them to the `#:linux-modules` to the `initrd` function.
  * Modify the `kernel-module-options-service-type` to pass in options via kernel command line always, maybe.

Then ZFS module can get loaded early, at boot, before the switch from `initrd` root to the "real" root.

* Create a service type that gathers additional pre-mount actions, which `raw-initrd` will accept as a new key `#:additional-pre-mount` and append to the `pre-mount` it passes to `boot-system`.
  * The `zfs-service-type` would then extend this service type to pass in an action to perform a `zpool import -a -N`, which makes ZFS scan for devices containing ZFS pools.
* Somehow figure out a static build for the ZFS package, so we can use a statically-linked `zpool` in the above.
  * The `zfs-service-type` already inherits a ZFS package from the given `base-package`, I imagine it would be possible to make further inheritance which modifies the build to be static.  It does require an additional build, though.  Maybe an additional `root-on-zfs?` field in `zfs-configuration` can gate this, so that we don't add the static build and extend the boot script if `(not root-on-zfs?)` (i.e. use `kernel-module-configuration-service-type` if `(not root-on-zfs?)`, else use the new put-it-in-the-initrd service types).
* Modify the various checks elsewhere so that ZFS poolnames can be passed as the `device` field of `file-system` records.  See https://issues.guix.gnu.org/45643#3
  * Then the installer can be modified so that if ZFS is included with the installer's profile, it can look at ZFS pools and offer to install to a ZFS pool instead of a `/dev` partition directly, then add `(file-system (device "rootpool") (mountpoint "/") (type "zfs"))` and the appropriate `zfs-service-type` and etc. and now we get `/` on ZFS.

But that can come later, much much later, I just need ZFS, *any* ZFS, before RAID5 eats more of my homework.

Thanks
raid5atemyhomework


> > But I guess the ZFS Linux kernel module can't be built-in into the kernel
> > anyway.
> > But that's a special case--in general, it's very much possible to make modules
> > built-in.
>
> ZFScan be built-in to the kernel, Ubuntu does it. You can't distribute it like that (Ubuntu distributes it like that but presumably they have enough lawyers to muddy the waters so that they can get away with it), but as the documentation in this patch notes: the user has every right to do whatever they want on the machine they own, including build a Linux kernel that has ZFS built-in and run it, they just can't make that version available to somebody else.
>
> So to go whole-hog, we would have a service that replaces the kernel package and inserts kernel module sources in-tree somehow, then compiles Linux-libre on the user's machine. That would probably be a lot more painful to install ZFS with (the user has to recompile the whole kernel at each update of either kernel or ZFS, whereas with a kernel module the user has to recompile just the kernel module), so maybe kernel module is still better overall.
>
> > > -                         ;; You'd think we could've used kernel-module-loader-service-type,
> > >
> > >
> >
> > Definitely.
> >
> > > -                         ;; but the kernel-module-loader shepherd service is dependent on
> > >
> > >
> > > -                         ;; file-systems,
> > >
> > >
> >
> > Yes--but why is that dependent on 'file-systems ? Is it because it needs /proc ?
> > Or is it an oversight ? I would prefer to get rid of this dependency and then
> > use kernel-module-loader-service-type.
>
> Dunno --- one VM I tested, I removed the`zfs-scan-automount` shepherd service from the `file-systems` target, and the VM still wouldn't boot, claiming a stack overflow (the same error which I got when I was still trying to use kernel-module-loader-service-type here). Or maybe I just got confused with which VM was which, testing VMs wasn't a stress-free vacation. I just want ZFS, because MD RAID5 ate my homework, this is getting tiresome...
>
> One thing I notice about `kernel-module-loader-service-type` is that it's not instantiated in essential services, or indeed anywhere in Guix. A few services do extend it. But my very rough understanding is that if you're going to extend a service, it had better be instantiated once in the list of services.
>
> In particular I note that the documentation for `kernel-module-loader-service-type` shows an example where it uses `service` to program the `kernel-module-loader-service-type`, not `simple-service`. This suggests to me that `kernel-module-loader-service-type` is broken because it's not in the list of essential services but is extensible. Maybe. It's designed as an extensible service, but isn't instantiated at default. Maybe that's what really bit me and not the shepherd circular dependency loop? shrug
>
> > Also, this manual loading of kernel modules is not supposed to be the way to
> > do things in Linux. That a kernel module was compiled as a module is
> > an implementation detail--so Linux should (and usually does) automatically
> > load kernel modules the first time a device for them is accessed (after all,
> > how would user space know whether something is compiled as a module or
> > built-in--that would be too much to ask).
>
> So how do I get ZFS loaded? Note that the devices it targets are block devices and it needs to scan for block devices that are formatted for ZFS. Do other filesystems have some autoload rule?
>
> Thanks
> raid5atemyhomework






^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] [PATCH 3/4] gnu: Fix ZFS package.
  2021-01-06 15:56 ` [bug#45692] [PATCH 3/4] gnu: Fix ZFS package raid5atemyhomework via Guix-patches via
@ 2021-01-07  8:23   ` Danny Milosavljevic
  0 siblings, 0 replies; 13+ messages in thread
From: Danny Milosavljevic @ 2021-01-07  8:23 UTC (permalink / raw)
  Cc: raid5atemyhomework, 45692

[-- Attachment #1: Type: text/plain, Size: 116 bytes --]

Pushed this one patch that fixes package "zfs" to guix master as
commit 8d3184c5bcf1a590f59a701f0484ab5cec673188.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#42193] kernel-module-configuration-service for configuring kernel parameters
  2021-01-07  5:38       ` [bug#45692] " raid5atemyhomework via Guix-patches via
@ 2021-01-07  9:16         ` raid5atemyhomework via Guix-patches via
  0 siblings, 0 replies; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-07  9:16 UTC (permalink / raw)
  To: 45592, 42193; +Cc: Brice Waegeneire

Hi Danny and Brice,

Cross-posting to 42193 as well.

> > > See also https://issues.guix.info/issue/42193 for an earlier attempt (which
> > > is already very far--but it has a bug somewhere).

I think what 42193 WIP 2/6 is missing is this hunk from my own 45592 PATCH 1/4:

```diff
diff --git a/gnu/system.scm b/gnu/system.scm
index c284a18379..5c530f176e 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm@@ -625,26 +616,10 @@ possible (that is if there's a LINUX keyword argument in the build system)."
   "Return the basic entries of the 'system' directory of OS for use as the
 value of the SYSTEM-SERVICE-TYPE service."
   (let* ((locale  (operating-system-locale-directory os))
-         (kernel  (operating-system-kernel os))
          (hurd    (operating-system-hurd os))
-         (modules (operating-system-kernel-loadable-modules os))
-         (kernel  (if hurd
-                      kernel
-                      (profile
-                       (content (packages->manifest
-                                 (cons kernel
-                                       (map (lambda (module)
-                                              (if (package? module)
-                                                  (package-for-kernel kernel
-                                                                      module)
-                                                  module))
-                                            modules))))
-                       (hooks (list linux-module-database)))))
          (initrd  (and (not hurd) (operating-system-initrd-file os)))
          (params  (operating-system-boot-parameters-file os)))
-    `(("kernel" ,kernel)
-      ,@(if hurd `(("hurd" ,hurd)) '())
-      ("parameters" ,params)
+    `(("parameters" ,params)
       ,@(if initrd `(("initrd" ,initrd)) '())
       ("locale" ,locale))))   ;used by libc
```



I hope we can progress this towards a workable ZFS-on-Guix.

Thanks
raid5atemyhomework




^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] [PATCH 4/4] gnu: Add ZFS service.
  2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
  2021-01-06 19:41   ` [bug#45703] kernel-module-configuration-service for configuring kernel parameters Danny Milosavljevic
@ 2021-01-08 15:02   ` raid5atemyhomework via Guix-patches via
  2021-01-09  8:31     ` raid5atemyhomework via Guix-patches via
  1 sibling, 1 reply; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-08 15:02 UTC (permalink / raw)
  To: 45692

This new version is dependent on https://issues.guix.info/45723 and https://issues.guix.info/45722

New features:

* We can now delay pool importation after particular mapped devices are opened or file systems are mounted.  This is useful if you want to make a ZFS pool out of LUKS containers, or mount an encrypted ZFS dataset with the encryption key in another (hopefully also encrypted) filesystem.
* We can now put other non-ZFS filesystems inside ZVOLs, block devices provided by a ZFS pool, and have them mounted after ZFS has scanned pools and opened such ZVOLs for use by the ystem.

It's now possible to have a bunch of LUKS containers in a ZPOOL that provides a ZVOL that contains an EXT4 filesystem (though why would you do that, just use builtin ZFS encryption).

This uses my [PATCH 1/4] instead of https://issues.guix.info/42193#4.  Should I use the latter instead?  What would get ZFS on Guix faster?

I decided to remove ZFS parameter options, because I don't know whether I should wait for https://issues.guix.info/42193 and how long that would take.


From d0e095136cd471f083d92416f12ca22b47301f31 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 09:24:20 +0800
Subject: [PATCH 4/4] gnu: Add service to install ZFS.

* gnu/services/file-systems.scm: New file.
(zfs-service-type): New variable.
(<zfs-configuration>): New type.
(%zfs-zvol-dependency): New variable.
* gnu/local.mk: Add gnu/services/file-systems.scm.
* gnu/services/base.scm (dependency->shepherd-service-name): Export.
* doc/guix.texi (ZFS file system): New subsection.
---
 doc/guix.texi                 | 107 +++++++++++++++++++++
 gnu/local.mk                  |   1 +
 gnu/services/base.scm         |   4 +-
 gnu/services/file-systems.scm | 174 ++++++++++++++++++++++++++++++++++
 4 files changed, 285 insertions(+), 1 deletion(-)
 create mode 100644 gnu/services/file-systems.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index a31d355780..714274b8c9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13886,6 +13886,113 @@ a file system declaration such as:
 compress-force=zstd,space_cache=v2"))
 @end lisp

+@node ZFS file system
+@subsection ZFS file system
+
+The ZFS file system has a license incompatible with the Linux kernel,
+and thus cannot be distributed with the kernel. However, as a user
+you have the right to do whatever you want on your own hardware,
+including download the ZFS source code, compile it, link it to your
+own private copy of Linux, and run it. You can even use the Guix build
+system to automate this.
+
+As a large and complex kernel module, ZFS on Linux has to be compiled
+with a specific version of the kernel. Often even the latest ZFS
+package available cannot be compiled with the latest Linux kernel
+package provided by Guix System. Thus, installing the @code{zfs}
+package is likely to fail.
+
+Instead, you have to use an older long-term-support Linux kernel.
+Do not use @code{linux-libre-lts} as the latest long-term-support
+kernel might be too new for the ZFS package; instead, explicitly
+select the version number, like @code{linux-libre-5.4}.
+
+Then, you have to modify your system configuration file and use the
+selected older kernel, and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+   ; @dots{}
+   linux)
+(use-service-modules
+   ; @dots{}
+   file-systems)
+
+(define my-kernel linux-libre-5.4)
+
+(operating-system
+  (kernel my-kernel)
+  ;; @dots{}
+  (services
+    (cons* (service zfs-service-type
+             (zfs-configuration
+               (kernel my-kernel)))
+           ; @dots{}
+           %desktop-services))
+  ;; @dots{}
+  )
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type of the service to compile and install OpenZFS to
+your operating system. It loads the ZFS module at startup, imports
+pools, mounts automounted ZFS datasets, installs the ZFS command
+line tools, and can provide module options for the ZFS module.  Its
+value must be a @code{zfs-configuration} record (see below).
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+  (zfs-configuration
+    (kernel linux-libre-5.4)
+    (options '("zfs_arc_max=5000000000"))))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS service.
+The available fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux kernel to compile ZFS for. Required.
+
+@item @code{base-zfs} (default: @code{zfs})
+The ZFS package to use. It will be modified to use the indicated
+kernel.
+
+@item @code{options} (default: @code{'()})
+A list of string options to pass as options to the ZFS module.
+These will be put in a @file{/etc/modprobe.d/zfs.conf} file,
+for example setting this to @code{'("zfs_admin_snapshot=1"
+"zfs_trim_extent_bytes_min=0")} will create the following file:
+
+@example
+options zfs zfs_admin_snapshot=1 zfs_trim_extent_bytes_min=0
+@end example
+@end table
+@end deftp
+
+Once your system has been configured to include the ZFS service
+and you have restarted the system, you can manage ZFS pools and
+datasets with @code{zpool} and @code{zfs} commands.
+
+ZFS datasets with an appropriate @code{mountpoint} property will
+be automounted at startup after the root file system is started.
+Encrypted datasets that are automounted will cause boot to pause
+and prompt for the password to be provided on the console.
+
+It's possible to have a ZFS dataset as your @code{/home} by simply
+setting the @code{mountpoint} property. However, note that ZFS will
+refuse to mount over a non-empty directory, so if your root
+filesystem already has a non-empty @code{/home}, you should remove
+it and its contents, then restart the system.
+
+Having ZFS as a root filesystem or as @code{/boot} is not supported
+yet.
+
 @node Mapped Devices
 @section Mapped Devices

diff --git a/gnu/local.mk b/gnu/local.mk
index 1151d4642e..5aeb45c4c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -602,6 +602,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/services/dict.scm				\
   %D%/services/dns.scm				\
   %D%/services/docker.scm			\
+  %D%/services/file-systems.scm			\
   %D%/services/authentication.scm		\
   %D%/services/games.scm			\
   %D%/services/ganeti.scm			\
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 13cfb6a8a2..ef3680583b 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -188,7 +188,9 @@

             references-file

-            %base-services))
+            %base-services
+
+            dependency->shepherd-service-name))

 ;;; Commentary:
 ;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..9061ab9582
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,174 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+  #:use-module (gnu packages file-systems)
+  #:use-module (gnu services)
+  #:use-module (gnu services base)
+  #:use-module (gnu services linux)
+  #:use-module (gnu services shepherd)
+  #:use-module (gnu system mapped-devices)
+  #:use-module (guix gexp)
+  #:use-module (guix packages)
+  #:use-module (guix records)
+  #:export (zfs-service-type
+
+            zfs-configuration
+            zfs-configuration?
+            zfs-configuration-kernel
+            zfs-configuration-base-zfs
+            zfs-configuration-dependencies
+
+            %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+  zfs-configuration make-zfs-configuration zfs-configuration?
+  ; kernel you want to compile the base-zfs module for.
+  (kernel         zfs-configuration-kernel)
+  ; base package that will be compiled for the kernel
+  (base-zfs       zfs-configuration-base-zfs
+                  (default zfs))
+  ; list of <mapped-device> | <file-system> we should wait for,
+  ; before scanning for ZFS pools.
+  (dependencies   zfs-configuration-dependencies
+                  (default '())))
+
+;; This is a synthetic and unusable MAPPED-DEVICE; its only use
+;; is to be added as a (dependency ...) of some FILE-SYSTEM.
+(define %zfs-zvol-dependency
+  (mapped-device
+    (source '())
+    ;; The /* prevents naming conflict with non-ZFS device mappings,
+    ;; since it is not a valid name for mapped devices, and also
+    ;; implies "all zvols" in terms of globs.
+    (targets '("zvol/*"))
+    (type #f)))
+
+(define (make-zfs-package conf)
+  (let ((base-zfs (zfs-configuration-base-zfs conf))
+        (kernel   (zfs-configuration-kernel conf)))
+    (package
+      (inherit base-zfs)
+      (name (string-join (list (package-name base-zfs)
+                               "for"
+                               (package-name kernel)
+                               (package-version kernel)
+                               "version")
+                         "-"))
+      (arguments (cons* #:linux kernel (package-arguments base-zfs))))))
+
+(define (zfs-loadable-module conf)
+  (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+  (let* ((zfs-package     (make-zfs-package conf))
+         (zpool           (file-append zfs-package "/sbin/zpool"))
+         (zfs             (file-append zfs-package "/sbin/zfs"))
+         (zvol_wait       (file-append zfs-package "/bin/zvol_wait"))
+         (scheme-modules  `((srfi srfi-1)
+                            (srfi srfi-34)
+                            (srfi srfi-35)
+                            (rnrs io ports)
+                            ,@%default-modules)))
+
+    (define zfs-scan
+      (shepherd-service
+        (provision '(zfs-scan))
+        (documentation "Scans for ZFS pools.")
+        (requirement `(kernel-module-loader
+                       root-file-system
+                       ,@(map dependency->shepherd-service-name
+                              (zfs-configuration-dependencies conf))))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error importing zpools: ~a~%"
+                                      (condition-message c))
+                              #f))
+                     ; TODO: optionally use a cachefile, for systems with dozens or
+                     ; hundreds of devices.
+                     (invoke/quiet #$zpool "import" "-a" "-N"))))
+        (stop #~(const #t))))
+
+    (define device-mapping-zvol/*
+      (shepherd-service
+        (provision '(device-mapping-zvol/*))
+        (documentation "Waits for ZFS ZVOL devices to appear.")
+        (requirement '(zfs-scan))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error waiting for zvols: ~a~%"
+                                      (condition-message c))
+                              #f))
+                     (invoke/quiet #$zvol_wait))))
+        (stop #~(const #t))))
+
+    (define zfs-automount
+      (shepherd-service
+        (provision '(zfs-automount))
+        (documentation "Automounts ZFS datasets.")
+        (requirement '(zfs-scan))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error automounting zfs: ~a~$")
+                              #f))
+                     ; (current-output-port) is typically connected to /dev/klog,
+                     ; so redirect it to (current-error-port) so that user can see
+                     ; prompts for passphrases on console
+                     (with-output-to-port (current-error-port)
+                       (lambda ()
+                         (invoke #$zfs "mount" "-a" "-l"))))))
+        (stop #~(lambda _
+                  ;; make sure we don't keep any ZFS mountpoints busy.
+                  (chdir "/")
+                  ;; unmount everything.
+                  (invoke/quiet #$zfs "unmount" "-a" "-f")))))
+
+    (list zfs-scan
+          device-mapping-zvol/*
+          zfs-automount)))
+
+(define zfs-service-type
+  (service-type (name 'zfs)
+                (extensions
+                  (list
+                    ; install the kernel module
+                    (service-extension kernel-loadable-module-service-type
+                                       zfs-loadable-module)
+                    ; load the kernel module
+                    (service-extension kernel-module-loader-service-type
+                                       (const '("zfs")))
+                    ; scan ZFS pools, automount filesystem, wait for zvols.
+                    (service-extension shepherd-root-service-type
+                                       zfs-shepherd-services)
+                    ; make sure automount occurs before file-systems target is reached
+                    (service-extension file-systems-target-service-type
+                                       (const '(zfs-automount)))
+                    ; install ZFS management tools
+                    (service-extension profile-service-type
+                                       (compose list make-zfs-package))
+                    ; install ZFS udev rules
+                    (service-extension udev-service-type
+                                       (compose list make-zfs-package))))
+                (description
+                  "Install ZFS, an advanced filesystem and volume manager.")))
--
2.30.0





^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules.
  2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
@ 2021-01-08 16:16   ` raid5atemyhomework via Guix-patches via
  0 siblings, 0 replies; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-08 16:16 UTC (permalink / raw)
  To: 45692

Is this patch acceptable? https://lists.nongnu.org/archive/html/guix-devel/2021-01/msg00070.html I reported that I compared a `guix system build` result of an `operating-system` that used the existing `(kernel-loadable-modules ...)` field, with and without this patch.

The resulting builds resulted in different hashes, but with exactly the same contents in the build --- `diff -r` very quickly reported no differences because it saw that nearly all the symlinks pointed to the same gnu store items.






^ permalink raw reply	[flat|nested] 13+ messages in thread

* [bug#45692] [PATCH 4/4] gnu: Add ZFS service.
  2021-01-08 15:02   ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
@ 2021-01-09  8:31     ` raid5atemyhomework via Guix-patches via
  0 siblings, 0 replies; 13+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-09  8:31 UTC (permalink / raw)
  To: 45692

New version again.  Change: Fix documentation `doc/guix.texi` to remove mentions of `options` and add documentation about the new `dependencies` features.


From dfe9ad7512d348933beb5b42041965ff604422b5 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 09:24:20 +0800
Subject: [PATCH 1/2] gnu: Add service to install ZFS.

* gnu/services/file-systems.scm: New file.
(zfs-service-type): New variable.
(<zfs-configuration>): New type.
(%zfs-zvol-dependency): New variable.
* gnu/local.mk: Add gnu/services/file-systems.scm.
* gnu/services/base.scm (dependency->shepherd-service-name): Export.
* doc/guix.texi (ZFS file system): New subsection.
---
 doc/guix.texi                 | 129 +++++++++++++++++++++++++
 gnu/local.mk                  |   1 +
 gnu/services/base.scm         |   4 +-
 gnu/services/file-systems.scm | 174 ++++++++++++++++++++++++++++++++++
 4 files changed, 307 insertions(+), 1 deletion(-)
 create mode 100644 gnu/services/file-systems.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index a31d355780..7004efe3c4 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13886,6 +13886,135 @@ a file system declaration such as:
 compress-force=zstd,space_cache=v2"))
 @end lisp

+@node ZFS file system
+@subsection ZFS file system
+
+The ZFS file system has a license incompatible with the Linux kernel,
+and thus cannot be distributed with the kernel. However, as a user
+you have the right to do whatever you want on your own hardware,
+including download the ZFS source code, compile it, link it to your
+own private copy of Linux, and run it. You can even use the Guix build
+system to automate this.
+
+As a large and complex kernel module, ZFS on Linux has to be compiled
+with a specific version of the kernel. Often even the latest ZFS
+package available cannot be compiled with the latest Linux kernel
+package provided by Guix System. Thus, installing the @code{zfs}
+package is likely to fail.
+
+Instead, you have to use an older long-term-support Linux kernel.
+Do not use @code{linux-libre-lts}, since the latest long-term-support
+kernel might be too new for the ZFS package; instead, explicitly
+select the version number, like @code{linux-libre-5.4}, and upgrade
+it manually later when you have verified that the ZFS version
+available on Guix can be compiled with a later LTS kernel.
+
+Then, you have to modify your system configuration file and use the
+selected older kernel, and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+   ; @dots{}
+   linux)
+(use-service-modules
+   ; @dots{}
+   file-systems)
+
+(define my-kernel linux-libre-5.4)
+
+(operating-system
+  (kernel my-kernel)
+  ;; @dots{}
+  (services
+    (cons* (service zfs-service-type
+             (zfs-configuration
+               (kernel my-kernel)))
+           ; @dots{}
+           %desktop-services))
+  ;; @dots{}
+  )
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type of the service to compile and install OpenZFS to
+your operating system. It loads the ZFS module at startup, imports
+pools, mounts automounted ZFS datasets, and installs the ZFS command
+line tools.  Its value must be a @code{zfs-configuration} record
+(see below).
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+  (zfs-configuration
+    (kernel linux-libre-5.4)))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS service.
+The available fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux kernel to compile ZFS for. Required. It
+@emph{must} be the same kernel you use in your operating system.
+
+@item @code{base-zfs} (default: @code{zfs})
+The ZFS package to use. It will be modified to use the indicated
+kernel.
+
+@item @code{dependencies} (default: @code{'()})
+A list of @code{<file-system>}s or @code{<mapped-device>}s that
+must be mounted or opened before ZFS scans for pools to import.
+For example, you might have LUKS containers as the leaf VDEVs of
+a ZFS pool.
+@end table
+@end deftp
+
+Once your system has been configured to include the ZFS service
+and you have restarted the system, you can manage ZFS pools and
+datasets with @code{zpool} and @code{zfs} commands.
+
+ZFS datasets with an appropriate @code{mountpoint} property will
+be automounted at startup after the root file system is started.
+Encrypted datasets that are automounted will cause boot to pause
+and prompt for the password to be provided on the console.
+
+It's possible to have a ZFS dataset as your @code{/home} by simply
+setting the @code{mountpoint} ZFS property. However, note that ZFS
+will refuse to mount over a non-empty directory, so if your root
+filesystem already has a non-empty @code{/home}, you should remove
+it and its contents, then restart the system.
+
+ZFS features @dfn{ZVOL}s, which are block devices that ZFS exposes
+to the system. You can put any file system inside a ZVOL. In order
+to mount such a filesystem at boot, you can declare it as dependent
+on @code{%zfs-zvol-dependency}.
+
+@defvr {Scheme Variable} %zfs-zvol-dependency
+This is an artificial @code{<mapped-device>} object which tells
+the file mounting service to wait for ZFS to provide ZVOLs before
+mounting the file system that is dependent on it. For example:
+
+@lisp
+(file-system
+  (device "/dev/zvol/pool-name/zvol-name")
+  (mount-point "/ext4-on-zfs")
+  (type "ext4")
+  (dependencies (list %zfs-zvol-dependency)))
+@end lisp
+
+Do @emph{not} add @code{%zfs-zvol-dependency} to your
+@code{mapped-devices} declaration, In addition, only use it as a
+@code{file-system} dependency if you instantiate a
+@code{zfs-service-type} service in your operating system.
+@end defvr
+
+Having ZFS as a root filesystem or as @code{/boot} is not supported
+yet.
+
 @node Mapped Devices
 @section Mapped Devices

diff --git a/gnu/local.mk b/gnu/local.mk
index 1151d4642e..5aeb45c4c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -602,6 +602,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/services/dict.scm				\
   %D%/services/dns.scm				\
   %D%/services/docker.scm			\
+  %D%/services/file-systems.scm			\
   %D%/services/authentication.scm		\
   %D%/services/games.scm			\
   %D%/services/ganeti.scm			\
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 13cfb6a8a2..ef3680583b 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -188,7 +188,9 @@

             references-file

-            %base-services))
+            %base-services
+
+            dependency->shepherd-service-name))

 ;;; Commentary:
 ;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..9061ab9582
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,174 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+  #:use-module (gnu packages file-systems)
+  #:use-module (gnu services)
+  #:use-module (gnu services base)
+  #:use-module (gnu services linux)
+  #:use-module (gnu services shepherd)
+  #:use-module (gnu system mapped-devices)
+  #:use-module (guix gexp)
+  #:use-module (guix packages)
+  #:use-module (guix records)
+  #:export (zfs-service-type
+
+            zfs-configuration
+            zfs-configuration?
+            zfs-configuration-kernel
+            zfs-configuration-base-zfs
+            zfs-configuration-dependencies
+
+            %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+  zfs-configuration make-zfs-configuration zfs-configuration?
+  ; kernel you want to compile the base-zfs module for.
+  (kernel         zfs-configuration-kernel)
+  ; base package that will be compiled for the kernel
+  (base-zfs       zfs-configuration-base-zfs
+                  (default zfs))
+  ; list of <mapped-device> | <file-system> we should wait for,
+  ; before scanning for ZFS pools.
+  (dependencies   zfs-configuration-dependencies
+                  (default '())))
+
+;; This is a synthetic and unusable MAPPED-DEVICE; its only use
+;; is to be added as a (dependency ...) of some FILE-SYSTEM.
+(define %zfs-zvol-dependency
+  (mapped-device
+    (source '())
+    ;; The /* prevents naming conflict with non-ZFS device mappings,
+    ;; since it is not a valid name for mapped devices, and also
+    ;; implies "all zvols" in terms of globs.
+    (targets '("zvol/*"))
+    (type #f)))
+
+(define (make-zfs-package conf)
+  (let ((base-zfs (zfs-configuration-base-zfs conf))
+        (kernel   (zfs-configuration-kernel conf)))
+    (package
+      (inherit base-zfs)
+      (name (string-join (list (package-name base-zfs)
+                               "for"
+                               (package-name kernel)
+                               (package-version kernel)
+                               "version")
+                         "-"))
+      (arguments (cons* #:linux kernel (package-arguments base-zfs))))))
+
+(define (zfs-loadable-module conf)
+  (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+  (let* ((zfs-package     (make-zfs-package conf))
+         (zpool           (file-append zfs-package "/sbin/zpool"))
+         (zfs             (file-append zfs-package "/sbin/zfs"))
+         (zvol_wait       (file-append zfs-package "/bin/zvol_wait"))
+         (scheme-modules  `((srfi srfi-1)
+                            (srfi srfi-34)
+                            (srfi srfi-35)
+                            (rnrs io ports)
+                            ,@%default-modules)))
+
+    (define zfs-scan
+      (shepherd-service
+        (provision '(zfs-scan))
+        (documentation "Scans for ZFS pools.")
+        (requirement `(kernel-module-loader
+                       root-file-system
+                       ,@(map dependency->shepherd-service-name
+                              (zfs-configuration-dependencies conf))))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error importing zpools: ~a~%"
+                                      (condition-message c))
+                              #f))
+                     ; TODO: optionally use a cachefile, for systems with dozens or
+                     ; hundreds of devices.
+                     (invoke/quiet #$zpool "import" "-a" "-N"))))
+        (stop #~(const #t))))
+
+    (define device-mapping-zvol/*
+      (shepherd-service
+        (provision '(device-mapping-zvol/*))
+        (documentation "Waits for ZFS ZVOL devices to appear.")
+        (requirement '(zfs-scan))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error waiting for zvols: ~a~%"
+                                      (condition-message c))
+                              #f))
+                     (invoke/quiet #$zvol_wait))))
+        (stop #~(const #t))))
+
+    (define zfs-automount
+      (shepherd-service
+        (provision '(zfs-automount))
+        (documentation "Automounts ZFS datasets.")
+        (requirement '(zfs-scan))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error automounting zfs: ~a~$")
+                              #f))
+                     ; (current-output-port) is typically connected to /dev/klog,
+                     ; so redirect it to (current-error-port) so that user can see
+                     ; prompts for passphrases on console
+                     (with-output-to-port (current-error-port)
+                       (lambda ()
+                         (invoke #$zfs "mount" "-a" "-l"))))))
+        (stop #~(lambda _
+                  ;; make sure we don't keep any ZFS mountpoints busy.
+                  (chdir "/")
+                  ;; unmount everything.
+                  (invoke/quiet #$zfs "unmount" "-a" "-f")))))
+
+    (list zfs-scan
+          device-mapping-zvol/*
+          zfs-automount)))
+
+(define zfs-service-type
+  (service-type (name 'zfs)
+                (extensions
+                  (list
+                    ; install the kernel module
+                    (service-extension kernel-loadable-module-service-type
+                                       zfs-loadable-module)
+                    ; load the kernel module
+                    (service-extension kernel-module-loader-service-type
+                                       (const '("zfs")))
+                    ; scan ZFS pools, automount filesystem, wait for zvols.
+                    (service-extension shepherd-root-service-type
+                                       zfs-shepherd-services)
+                    ; make sure automount occurs before file-systems target is reached
+                    (service-extension file-systems-target-service-type
+                                       (const '(zfs-automount)))
+                    ; install ZFS management tools
+                    (service-extension profile-service-type
+                                       (compose list make-zfs-package))
+                    ; install ZFS udev rules
+                    (service-extension udev-service-type
+                                       (compose list make-zfs-package))))
+                (description
+                  "Install ZFS, an advanced filesystem and volume manager.")))
--
2.30.0




^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2021-01-09  8:32 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
2021-01-08 16:16   ` raid5atemyhomework via Guix-patches via
2021-01-06 15:55 ` [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services raid5atemyhomework via Guix-patches via
2021-01-06 15:56 ` [bug#45692] [PATCH 3/4] gnu: Fix ZFS package raid5atemyhomework via Guix-patches via
2021-01-07  8:23   ` Danny Milosavljevic
2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
2021-01-06 19:41   ` [bug#45703] kernel-module-configuration-service for configuring kernel parameters Danny Milosavljevic
2021-01-07  0:04     ` raid5atemyhomework via Guix-patches via
2021-01-07  5:38       ` [bug#45692] " raid5atemyhomework via Guix-patches via
2021-01-07  9:16         ` [bug#42193] " raid5atemyhomework via Guix-patches via
2021-01-08 15:02   ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
2021-01-09  8:31     ` raid5atemyhomework via Guix-patches via

unofficial mirror of guix-patches@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-patches/1 guix-patches/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-patches guix-patches/ https://yhetil.org/guix-patches \
		guix-patches@gnu.org
	public-inbox-index guix-patches

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.patches


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git