* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
@ 2021-01-06 15:52 raid5atemyhomework via Guix-patches via
2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
` (12 more replies)
0 siblings, 13 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:52 UTC (permalink / raw)
To: 45692
Hi all,
Sorry for the excessive number of patches.
This patchset contains 4 patches, two that add new features to Guix and do not involve ZFS at all, one bugfix of the ZFS package, and one which makes installing ZFS on Guix easy.
* Patch 1: Create an extensible service type that installs kernel-loadable modules into the kernel.
* This allows a ZFS service to add the ZFS module to the kernel.
* Patch 2: Make the `file-systems` shepherd service an extensible target, so that other services can add more requirements for `file-systems`.
* This allows a ZFS service to make `file-systems` wait for the ZFS automount at startup, which is necessary in any case in order to ensure that ZFS has scanned for ZFS pools installed in the hardware.
* Patch 3: Fix the ZFS package definition.
* Patch 4: Add a ZFS service type that installs everything needed for basic ZFS usage!
I've tested the ZFS functionality in about a half-dozen VMs already, including checking that the ZFS setup persists across reboots of the VM:
* No ZFS installed.
* With ZFS installed but without any ZFS pools.
* With ZFS pool that is automounted.
* With ZFS pool in legacy mode (needs manual `mount -t zfs poolname /mountpoint`).
* Encrypted ZFS dataset (prompts passphrase at boot to mount).
* Encrypted ZFS dataseet as `/home` (prompts passphrase at boot to mount, user `/home` directories are created and maintained).
Other bits of ZFS functionality would be *nice* but aren't in this patchset yet (and at least for my purposes are not really necessary, except perhaps the auto-scrubbing, I just need to wrap my head around `mcron`...):
* Auto-scrubbing of ZFS pools to ensure no data corruption.
* Auto-snapshotting of ZFS datasets to protect against user error.
* `/` on ZFS.
* `/boot` on ZFS.
* Managing ZFS "legacy" mountpoints on `(file-system ...)` declarations.
* Managing ZFS pool setup in the `operating-system` declaration somehow (not sure how to do this).
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
@ 2021-01-06 15:54 ` raid5atemyhomework via Guix-patches via
2021-01-08 16:16 ` raid5atemyhomework via Guix-patches via
` (2 more replies)
2021-01-06 15:55 ` [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services raid5atemyhomework via Guix-patches via
` (11 subsequent siblings)
12 siblings, 3 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:54 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
From 4beb73c62995cf236b402dad8e1c36016027c781 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH 1/4] gnu: Allow services to install kernel-loadable modules.
* gnu/system.scm (operating-system-directory-base-entries): Remove code
to handle generation of "kernel" and "hurd".
(operating-system-default-essential-services): Instantiate
kernel-loadable-module-service.
(hurd-default-essential-services): Instantiate
kernel-loadable-module-service.
(package-for-kernel): Move ...
* gnu/services.scm: ... to here.
(kernel-loadable-module-service-type): New variable.
(kernel-loadable-module-service): New procedure.
* gnu/tests/linux-modules.scm (run-loadable-kernel-modules-test): Move
code to ...
(run-loadable-kernel-modules-test-base): ... new procedure here.
(run-loadable-kernel-modules-service-test): New procedure.
(%test-loadable-kernel-modules-service-0): New variable.
(%test-loadable-kernel-modules-service-1): New variable.
(%test-loadable-kernel-modules-service-2): New variable.
* doc/guix.texi: Document kernel-loadable-module-service-type.
---
doc/guix.texi | 6 +++
gnu/services.scm | 70 ++++++++++++++++++++++++++++++++
gnu/system.scm | 37 +++++------------
gnu/tests/linux-modules.scm | 81 ++++++++++++++++++++++++++++++++-----
4 files changed, 157 insertions(+), 37 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 0f6e95a65a..78770151e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32409,6 +32409,12 @@ configuration when you use @command{guix system reconfigure},
@command{guix system init}, or @command{guix deploy}.
@end defvr
+@defvr {Scheme Variable} kernel-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+@end defvr
+
@node Shepherd Services
@subsection Shepherd Services
diff --git a/gnu/services.scm b/gnu/services.scm
index 13259dfaee..d7332a46b2 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2016 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -33,6 +34,8 @@
#:use-module (guix diagnostics)
#:autoload (guix openpgp) (openpgp-format-fingerprint)
#:use-module (guix modules)
+ #:use-module (guix packages)
+ #:use-module (guix utils)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages hurd)
@@ -75,6 +78,7 @@
service-back-edges
instantiate-missing-services
fold-services
+ kernel-loadable-module-service
service-error?
missing-value-service-error?
@@ -106,6 +110,7 @@
profile-service-type
firmware-service-type
gc-root-service-type
+ kernel-loadable-module-service-type
%boot-service
%activation-service
@@ -864,6 +869,71 @@ as Wifi cards.")))
will not be reclaimed by the garbage collector.")
(default-value '())))
+;; Configuration for the kernel builder.
+(define-record-type* <kernel-builder-configuration> kernel-builder-configuration
+ make-kernel-builder-configuration
+ kernel-builder-configuration?
+ this-kernel-builder-configuration
+
+ (kernel kernel-builder-configuration-kernel (default #f))
+ (hurd kernel-builder-configuration-hurd (default #f))
+ (modules kernel-builder-configuration-modules (default '())))
+
+(define (package-for-kernel target-kernel module-package)
+ "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
+possible (that is if there's a LINUX keyword argument in the build system)."
+ (package
+ (inherit module-package)
+ (arguments
+ (substitute-keyword-arguments (package-arguments module-package)
+ ((#:linux kernel #f)
+ target-kernel)))))
+
+(define (kernel-builder-configuration->system-entry config)
+ "Return the kernel and hurd entries of the 'system' directory."
+ (mbegin %store-monad
+ (let* ((kernel (kernel-builder-configuration-kernel config))
+ (hurd (kernel-builder-configuration-hurd config))
+ (modules (kernel-builder-configuration-modules config))
+ (kernel (if hurd
+ kernel
+ (profile
+ (content (packages->manifest
+ (cons kernel
+ (map (lambda (module)
+ (if (package? module)
+ (package-for-kernel kernel module)
+ module))
+ modules))))
+ (hooks (list linux-module-database))))))
+ (return `(("kernel" ,kernel)
+ ,@(if hurd `(("hurd" ,hurd)) '()))))))
+
+(define (kernel-builder-configuration-add-modules config modules)
+ "Constructs a kernel builder configuration that has its modules extended."
+ (kernel-builder-configuration
+ (inherit config)
+ (modules (append (kernel-builder-configuration-modules config) modules))))
+
+(define kernel-loadable-module-service-type
+ (service-type (name 'kernel-loadable-modules)
+ (extensions
+ (list (service-extension system-service-type
+ kernel-builder-configuration->system-entry)))
+ (compose concatenate)
+ (extend kernel-builder-configuration-add-modules)
+ (description
+ "Register packages containing kernel-loadable modules and adds them
+to the system.")))
+
+(define (kernel-loadable-module-service kernel hurd modules)
+ "Constructs the service that sets up kernel loadable modules."
+ (service kernel-loadable-module-service-type
+ (kernel-builder-configuration
+ (kernel kernel)
+ (hurd hurd)
+ (modules modules))))
+
\f
;;;
;;; Service folding.
diff --git a/gnu/system.scm b/gnu/system.scm
index c284a18379..5c530f176e 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -12,6 +12,7 @@
;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <jannek@gnu.org>
;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -600,16 +601,6 @@ OS."
(file-append (operating-system-kernel os)
"/" (system-linux-image-file-name))))
-(define (package-for-kernel target-kernel module-package)
- "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
-possible (that is if there's a LINUX keyword argument in the build system)."
- (package
- (inherit module-package)
- (arguments
- (substitute-keyword-arguments (package-arguments module-package)
- ((#:linux kernel #f)
- target-kernel)))))
-
(define %default-modprobe-blacklist
;; List of kernel modules to blacklist by default.
'("usbmouse" ;races with bcm5974, see <https://bugs.gnu.org/35574>
@@ -625,26 +616,10 @@ possible (that is if there's a LINUX keyword argument in the build system)."
"Return the basic entries of the 'system' directory of OS for use as the
value of the SYSTEM-SERVICE-TYPE service."
(let* ((locale (operating-system-locale-directory os))
- (kernel (operating-system-kernel os))
(hurd (operating-system-hurd os))
- (modules (operating-system-kernel-loadable-modules os))
- (kernel (if hurd
- kernel
- (profile
- (content (packages->manifest
- (cons kernel
- (map (lambda (module)
- (if (package? module)
- (package-for-kernel kernel
- module)
- module))
- modules))))
- (hooks (list linux-module-database)))))
(initrd (and (not hurd) (operating-system-initrd-file os)))
(params (operating-system-boot-parameters-file os)))
- `(("kernel" ,kernel)
- ,@(if hurd `(("hurd" ,hurd)) '())
- ("parameters" ,params)
+ `(("parameters" ,params)
,@(if initrd `(("initrd" ,initrd)) '())
("locale" ,locale)))) ;used by libc
@@ -663,6 +638,10 @@ bookkeeping."
(host-name (host-name-service (operating-system-host-name os)))
(entries (operating-system-directory-base-entries os)))
(cons* (service system-service-type entries)
+ (kernel-loadable-module-service
+ (operating-system-kernel os)
+ (operating-system-hurd os)
+ (operating-system-kernel-loadable-modules os))
%boot-service
;; %SHEPHERD-ROOT-SERVICE must come last so that the gexp that
@@ -699,6 +678,10 @@ bookkeeping."
(define (hurd-default-essential-services os)
(let ((entries (operating-system-directory-base-entries os)))
(list (service system-service-type entries)
+ (kernel-loadable-module-service
+ (operating-system-kernel os)
+ (operating-system-hurd os)
+ (operating-system-kernel-loadable-modules os))
%boot-service
%hurd-startup-service
%activation-service
diff --git a/gnu/tests/linux-modules.scm b/gnu/tests/linux-modules.scm
index 953b132ef7..9739e4124d 100644
--- a/gnu/tests/linux-modules.scm
+++ b/gnu/tests/linux-modules.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2019 Jakob L. Kreuze <zerodaysfordays@sdf.org>
;;; Copyright © 2020 Danny Milosavljevic <dannym@scratchpost.org>
;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -34,7 +35,10 @@
#:use-module (guix utils)
#:export (%test-loadable-kernel-modules-0
%test-loadable-kernel-modules-1
- %test-loadable-kernel-modules-2))
+ %test-loadable-kernel-modules-2
+ %test-loadable-kernel-modules-service-0
+ %test-loadable-kernel-modules-service-1
+ %test-loadable-kernel-modules-service-2))
;;; Commentary:
;;;
@@ -66,17 +70,11 @@ that MODULES are actually loaded."
(member module modules string=?))
'#$modules))))))
-(define* (run-loadable-kernel-modules-test module-packages module-names)
- "Run a test of an OS having MODULE-PACKAGES, and verify that MODULE-NAMES
-are loaded in memory."
+(define* (run-loadable-kernel-modules-test-base base-os module-names)
+ "Run a test of BASE-OS, verifying that MODULE-NAMES are loaded in memory."
(define os
(marionette-operating-system
- (operating-system
- (inherit (simple-operating-system))
- (services (cons (service kernel-module-loader-service-type module-names)
- (operating-system-user-services
- (simple-operating-system))))
- (kernel-loadable-modules module-packages))
+ base-os
#:imported-modules '((guix combinators))))
(define vm (virtual-machine os))
(define (test script)
@@ -98,6 +96,37 @@ are loaded in memory."
(gexp->derivation "loadable-kernel-modules"
(test (modules-loaded?-program os module-names))))
+(define* (run-loadable-kernel-modules-test module-packages module-names)
+ "Run a test of an OS having MODULE-PACKAGES, and verify that MODULE-NAMES
+are loaded in memory."
+ (run-loadable-kernel-modules-test-base
+ (operating-system
+ (inherit (simple-operating-system))
+ (services (cons (service kernel-module-loader-service-type module-names)
+ (operating-system-user-services
+ (simple-operating-system))))
+ (kernel-loadable-modules module-packages))
+ module-names))
+
+(define* (run-loadable-kernel-modules-service-test module-packages module-names)
+ "Run a test of an OS having MODULE-PACKAGES, which are loaded by creating a
+service that extends KERNEL-LOADABLE-MODULE-SERVICE-TYPE. Then verify that
+MODULE-NAMES are loaded in memory."
+ (define module-installing-service-type
+ (service-type
+ (name 'module-installing-service)
+ (extensions (list (service-extension kernel-loadable-module-service-type
+ (const module-packages))))
+ (default-value #f)))
+ (run-loadable-kernel-modules-test-base
+ (operating-system
+ (inherit (simple-operating-system))
+ (services (cons* (service kernel-module-loader-service-type module-names)
+ (service module-installing-service-type)
+ (operating-system-user-services
+ (simple-operating-system)))))
+ module-names))
+
(define %test-loadable-kernel-modules-0
(system-test
(name "loadable-kernel-modules-0")
@@ -129,3 +158,35 @@ with two extra modules.")
(package-arguments
ddcci-driver-linux))))))
'("acpi_call" "ddcci")))))
+
+(define %test-loadable-kernel-modules-service-0
+ (system-test
+ (name "loadable-kernel-modules-service-0")
+ (description "Tests loadable kernel modules extensible service with no
+extra modules.")
+ (value (run-loadable-kernel-modules-service-test '() '()))))
+
+(define %test-loadable-kernel-modules-service-1
+ (system-test
+ (name "loadable-kernel-modules-service-1")
+ (description "Tests loadable kernel modules extensible service with one
+extra module.")
+ (value (run-loadable-kernel-modules-service-test
+ (list ddcci-driver-linux)
+ '("ddcci")))))
+
+(define %test-loadable-kernel-modules-service-2
+ (system-test
+ (name "loadable-kernel-modules-service-2")
+ (description "Tests loadable kernel modules extensible service with two
+extra modules.")
+ (value (run-loadable-kernel-modules-service-test
+ (list acpi-call-linux-module
+ (package
+ (inherit ddcci-driver-linux)
+ (arguments
+ `(#:linux #f
+ ,@(strip-keyword-arguments '(#:linux)
+ (package-arguments
+ ddcci-driver-linux))))))
+ '("acpi_call" "ddcci")))))
--
2.29.2
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
@ 2021-01-06 15:55 ` raid5atemyhomework via Guix-patches via
2021-01-23 13:05 ` 宋文武
2021-01-06 15:56 ` [bug#45692] [PATCH 3/4] gnu: Fix ZFS package raid5atemyhomework via Guix-patches via
` (10 subsequent siblings)
12 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:55 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
From 792a8f8efc95e4fe9a94d42f839ddcfb034b8540 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 08:15:54 +0800
Subject: [PATCH 2/4] gnu: Make file-systems target extensible by services.
* gnu/services/base.scm (file-system-shepherd-services): Move file-systems
shepherd service to ...
(file-systems-target-shepherd-services): ... new procedure here.
(file-systems-target-service-type): New variable.
(file-system-service-type): Extend file-systems-target service to add each
file-system as a requirement.
* gnu/system.scm (operating-system-default-essential-services): Instantiate
file-systems-target-service-type.
(hurd-default-essential-services): Instantiate file-systems-target-service-type.
---
gnu/services/base.scm | 37 +++++++++++++++++++++++++++----------
gnu/system.scm | 2 ++
2 files changed, 29 insertions(+), 10 deletions(-)
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 945b546607..13cfb6a8a2 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -13,6 +13,7 @@
;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;; Copyright © 2020 Florian Pelz <pelzflorian@pelzflorian.de>
;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -67,6 +68,7 @@
#:export (fstab-service-type
root-file-system-service
file-system-service-type
+ file-systems-target-service-type
swap-service
host-name-service
console-keymap-service
@@ -362,18 +364,29 @@ FILE-SYSTEM."
(gnu system file-systems)
,@%default-modules)))))))
+(define (file-systems-target-shepherd-services requirements)
+ (list
+ (shepherd-service
+ (provision '(file-systems))
+ (requirement (cons* 'root-file-system 'user-file-systems requirements))
+ (documentation "Target for all the initially-mounted file systems")
+ (start #~(const #t))
+ (stop #~(const #t)))))
+(define file-systems-target-service-type
+ (service-type
+ (name 'file-systems)
+ (extensions (list (service-extension shepherd-root-service-type
+ file-systems-target-shepherd-services)))
+ (compose concatenate)
+ (extend append)
+ ;; Extensions can add new values to this list.
+ (default-value '())
+ (description "The @code{file-systems} service is the target that is started
+when all file systems have been mounted.")))
+
(define (file-system-shepherd-services file-systems)
"Return the list of Shepherd services for FILE-SYSTEMS."
(let* ((file-systems (filter file-system-mount? file-systems)))
- (define sink
- (shepherd-service
- (provision '(file-systems))
- (requirement (cons* 'root-file-system 'user-file-systems
- (map file-system->shepherd-service-name
- file-systems)))
- (documentation "Target for all the initially-mounted file systems")
- (start #~(const #t))
- (stop #~(const #f))))
(define known-mount-points
(map file-system-mount-point file-systems))
@@ -403,7 +416,7 @@ FILE-SYSTEM."
(filter (negate known?) (mount-points)))
#f))))
- (cons* sink user-unmount
+ (cons* user-unmount
(map file-system-shepherd-service file-systems))))
(define (file-system-fstab-entries file-systems)
@@ -431,6 +444,10 @@ FILE-SYSTEM."
(service-extension fstab-service-type
file-system-fstab-entries)
+ ;; Have 'file-systems' depend on each declared file system.
+ (service-extension file-systems-target-service-type
+ (cut map file-system->shepherd-service-name <>))
+
;; Have 'user-processes' depend on 'file-systems'.
(service-extension user-processes-service-type
(const '(file-systems)))))
diff --git a/gnu/system.scm b/gnu/system.scm
index 5c530f176e..6987641ee8 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -667,6 +667,7 @@ bookkeeping."
(operating-system-setuid-programs os))
(service profile-service-type
(operating-system-packages os))
+ (service file-systems-target-service-type)
other-fs
(append mappings swaps
@@ -691,6 +692,7 @@ bookkeeping."
(operating-system-groups os))
(operating-system-skeletons os))
(root-file-system-service)
+ (service file-systems-target-service-type)
(service file-system-service-type '())
(service fstab-service-type
(filter file-system-needed-for-boot?
--
2.29.2
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 3/4] gnu: Fix ZFS package.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
2021-01-06 15:55 ` [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services raid5atemyhomework via Guix-patches via
@ 2021-01-06 15:56 ` raid5atemyhomework via Guix-patches via
2021-01-07 8:23 ` Danny Milosavljevic
2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
` (9 subsequent siblings)
12 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:56 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
From d1c88161f2a32f4c03de57ee9d7582cc6c40f6bc Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Tue, 5 Jan 2021 20:00:50 +0800
Subject: [PATCH 3/4] gnu: Fix ZFS package.
* gnu/packages/file-systems.scm (zfs): Correct "util-linux" input, add
separate "util-linux:lib" input.
---
gnu/packages/file-systems.scm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 895ad069c5..2c5ad95d63 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -6,6 +6,7 @@
;;; Copyright © 2019, 2020 Efraim Flashner <efraim@flashner.co.il>
;;; Copyright © 2020 Raghav Gururajan <raghavgururajan@disroot.org>
;;; Copyright © 2020 Morgan Smith <Morgan.J.Smith@outlook.com>
+;;; Copyright © 2021 raid5atemyhoemwork <raid5atemyhomework@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -944,7 +945,8 @@ APFS.")
("openssl" ,openssl)
("python" ,python)
("python-cffi" ,python-cffi)
- ("util-linux" ,util-linux "lib")
+ ("util-linux" ,util-linux)
+ ("util-linux:lib" ,util-linux "lib")
("zlib" ,zlib)))
(home-page "https://zfsonlinux.org/")
(synopsis "Native ZFS on Linux")
--
2.29.2
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 4/4] gnu: Add ZFS service.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (2 preceding siblings ...)
2021-01-06 15:56 ` [bug#45692] [PATCH 3/4] gnu: Fix ZFS package raid5atemyhomework via Guix-patches via
@ 2021-01-06 15:57 ` raid5atemyhomework via Guix-patches via
2021-01-06 19:41 ` [bug#45703] kernel-module-configuration-service for configuring kernel parameters Danny Milosavljevic
2021-01-08 15:02 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
2021-03-22 14:33 ` [bug#45692] [PATCH v3 0/3] New patch series for " raid5atemyhomework via Guix-patches via
` (8 subsequent siblings)
12 siblings, 2 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-06 15:57 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
From deded39fe88c44a18b0b66bab8c4300aca4c387e Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 09:24:20 +0800
Subject: [PATCH 4/4] gnu: Add ZFS service.
* gnu/services/file-systems.scm: New file.
(zfs-service-type): New variable.
(zfs-configuration): New variable.
(zfs-configuration?): New procedure.
* gnu/local.mk: Add gnu/services/file-systems.scm.
* doc/guix.texi (ZFS file system): New subsection.
---
doc/guix.texi | 107 ++++++++++++++++++++++++++
gnu/local.mk | 2 +
gnu/services/file-systems.scm | 136 ++++++++++++++++++++++++++++++++++
3 files changed, 245 insertions(+)
create mode 100644 gnu/services/file-systems.scm
diff --git a/doc/guix.texi b/doc/guix.texi
index 78770151e3..62437083cb 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13885,6 +13885,113 @@ a file system declaration such as:
compress-force=zstd,space_cache=v2"))
@end lisp
+@node ZFS file system
+@subsection ZFS file system
+
+The ZFS file system has a license incompatible with the Linux kernel,
+and thus cannot be distributed with the kernel. However, as a user
+you have the right to do whatever you want on your own hardware,
+including download the ZFS source code, compile it, link it to your
+own private copy of Linux, and run it. You can even use the Guix build
+system to automate this.
+
+As a large and complex kernel module, ZFS on Linux has to be compiled
+with a specific version of the kernel. Often even the latest ZFS
+package available cannot be compiled with the latest Linux kernel
+package provided by Guix System. Thus, installing the @code{zfs}
+package is likely to fail.
+
+Instead, you have to use an older long-term-support Linux kernel.
+Do not use @code{linux-libre-lts} as the latest long-term-support
+kernel might be too new for the ZFS package; instead, explicitly
+select the version number, like @code{linux-libre-5.4}.
+
+Then, you have to modify your system configuration file and use the
+selected older kernel, and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+ ; @dots{}
+ linux)
+(use-service-modules
+ ; @dots{}
+ file-systems)
+
+(define my-kernel linux-libre-5.4)
+
+(operating-system
+ (kernel my-kernel)
+ ;; @dots{}
+ (services
+ (cons* (service zfs-service-type
+ (zfs-configuration
+ (kernel my-kernel)))
+ ; @dots{}
+ %desktop-services))
+ ;; @dots{}
+ )
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type of the service to compile and install OpenZFS to
+your operating system. It loads the ZFS module at startup, imports
+pools, mounts automounted ZFS datasets, installs the ZFS command
+line tools, and can provide module options for the ZFS module. Its
+value must be a @code{zfs-configuration} record (see below).
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+ (zfs-configuration
+ (kernel linux-libre-5.4)
+ (options '("zfs_arc_max=5000000000"))))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS service.
+The available fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux kernel to compile ZFS for. Required.
+
+@item @code{base-zfs} (default: @code{zfs})
+The ZFS package to use. It will be modified to use the indicated
+kernel.
+
+@item @code{options} (default: @code{'()})
+A list of string options to pass as options to the ZFS module.
+These will be put in a @file{/etc/modprobe.d/zfs.conf} file,
+for example setting this to @code{'("zfs_admin_snapshot=1"
+"zfs_trim_extent_bytes_min=0")} will create the following file:
+
+@example
+options zfs zfs_admin_snapshot=1 zfs_trim_extent_bytes_min=0
+@end example
+@end table
+@end deftp
+
+Once your system has been configured to include the ZFS service
+and you have restarted the system, you can manage ZFS pools and
+datasets with @code{zpool} and @code{zfs} commands.
+
+ZFS datasets with an appropriate @code{mountpoint} property will
+be automounted at startup after the root file system is started.
+Encrypted datasets that are automounted will cause boot to pause
+and prompt for the password to be provided on the console.
+
+It's possible to have a ZFS dataset as your @code{/home} by simply
+setting the @code{mountpoint} property. However, note that ZFS will
+refuse to mount over a non-empty directory, so if your root
+filesystem already has a non-empty @code{/home}, you should remove
+it and its contents, then restart the system.
+
+Having ZFS as a root filesystem or as @code{/boot} is not supported
+yet.
+
@node Mapped Devices
@section Mapped Devices
diff --git a/gnu/local.mk b/gnu/local.mk
index 53a67e6ffe..49073ac2ac 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -39,6 +39,7 @@
# Copyright © 2020 Martin Becze <mjbecze@riseup.net>
# Copyright © 2020 Malte Frank Gerdes <mate.f.gerdes@gmail.com>
# Copyright © 2020 Vinicius Monego <monego@posteo.net>
+# Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
#
# This file is part of GNU Guix.
#
@@ -601,6 +602,7 @@ GNU_SYSTEM_MODULES = \
%D%/services/dict.scm \
%D%/services/dns.scm \
%D%/services/docker.scm \
+ %D%/services/file-systems.scm \
%D%/services/authentication.scm \
%D%/services/games.scm \
%D%/services/ganeti.scm \
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..7a92a2b9d5
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,136 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+ #:use-module (gnu packages file-systems)
+ #:use-module (gnu services)
+ #:use-module (gnu services base)
+ #:use-module (gnu services shepherd)
+ #:use-module (guix gexp)
+ #:use-module (guix packages)
+ #:use-module (guix records)
+ #:export (zfs-service-type
+ zfs-configuration
+ zfs-configuration?))
+
+(define-record-type* <zfs-configuration>
+ zfs-configuration make-zfs-configuration zfs-configuration?
+ ; kernel you want to compile the base-zfs module for.
+ (kernel zfs-configuration-kernel)
+ ; base package that will be compiled for the kernel
+ (base-zfs zfs-configuration-base-zfs (default zfs))
+ ; list of string options.
+ (options zfs-configuration-options (default '())))
+
+(define (make-zfs-package conf)
+ (let ((base-zfs (zfs-configuration-base-zfs conf))
+ (kernel (zfs-configuration-kernel conf)))
+ (package
+ (inherit base-zfs)
+ (name (string-join (list (package-name base-zfs)
+ "for"
+ (package-name kernel)
+ (package-version kernel)
+ "version")
+ "-"))
+ (arguments (cons* #:linux kernel (package-arguments base-zfs))))))
+
+(define (zfs-loadable-module conf)
+ (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool")))
+ (list
+ (shepherd-service
+ (documentation "Scans for ZFS pools and automounts filesystems.")
+ (provision '(zfs-scan-automount))
+ (requirement '(root-file-system))
+ (modules `((srfi srfi-1)
+ (srfi srfi-34)
+ (srfi srfi-35)
+ (rnrs io ports)
+ ,@%default-modules))
+ (start #~(lambda _
+ (and
+ ;; You'd think we could've used kernel-module-loader-service-type,
+ ;; but the kernel-module-loader shepherd service is dependent on
+ ;; file-systems, and file-systems is made dependent on this
+ ;; service. And we need the kernel module to be loaded before we
+ ;; scan for ZFS pools. So break the dependency loop by just
+ ;; loading ZFS module here by ourselves.
+ (or (file-exists? "/proc/sys/kernel/modprobe")
+ (begin
+ (format (current-error-port) "error loading 'zfs' module: ~a~%"
+ "Kernel is missing loadable module support.")
+ #f))
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "error loading 'zfs' module: ~a~%"
+ (condition-message c))
+ #f))
+ (let ((modprobe (call-with-input-file "/proc/sys/kernel/modprobe"
+ get-line)))
+ (invoke/quiet modprobe "--" "zfs")))
+
+ ; scan for pools and automount contained datasets.
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "error importing zpools: ~a~%"
+ (condition-message?))
+ #f))
+ ;; (current-output-port) is typically connected to /dev/klog,
+ ;; so redirect it to (current-error-port) so that user can see
+ ;; prompts for passphrases on console
+ (with-output-to-port (current-error-port)
+ (lambda ()
+ (invoke #$zpool "import" "-a" "-l")))))))
+ (stop #~(const #t))))))
+
+(define (zfs-profile-service conf)
+ (list (make-zfs-package conf)))
+
+(define (zfs-etc-service conf)
+ (let ((options (zfs-configuration-options conf)))
+ (if (null? options)
+ '()
+ `(("modprobe.d/zfs.conf"
+ ,(plain-file "zfs.conf"
+ (string-join (cons "options zfs" options) " ")))))))
+
+(define zfs-service-type
+ (service-type (name 'zfs)
+ (extensions
+ (list
+ ; install the kernel module
+ (service-extension kernel-loadable-module-service-type
+ zfs-loadable-module)
+ ; load ZFS module, scan ZFS pools, and automount filesystems
+ (service-extension shepherd-root-service-type
+ zfs-shepherd-services)
+ ; make sure automount occurs before file-systems target is reached
+ (service-extension file-systems-target-service-type
+ (const '(zfs-scan-automount)))
+ ; install ZFS management tools
+ (service-extension profile-service-type
+ zfs-profile-service)
+ ; install ZFS module options
+ (service-extension etc-service-type
+ zfs-etc-service)))
+ (description
+ "Install ZFS, an advanced filesystem and volume manager.")))
--
2.29.2
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45703] kernel-module-configuration-service for configuring kernel parameters
2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
@ 2021-01-06 19:41 ` Danny Milosavljevic
2021-01-07 0:04 ` raid5atemyhomework via Guix-patches via
2021-01-08 15:02 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
1 sibling, 1 reply; 81+ messages in thread
From: Danny Milosavljevic @ 2021-01-06 19:41 UTC (permalink / raw)
Cc: raid5atemyhomework, 45703, 45692
[-- Attachment #1: Type: text/plain, Size: 5916 bytes --]
Hi raid5atemyhomework,
Hi everyone,
@raid5atemyhomework, thanks for all the patches!
On Wed, 06 Jan 2021 15:57:19 +0000
raid5atemyhomework via Guix-patches via <guix-patches@gnu.org> wrote:
> +@item @code{options} (default: @code{'()})
> +A list of string options to pass as options to the ZFS module.
> +These will be put in a @file{/etc/modprobe.d/zfs.conf} file,
> +for example setting this to @code{'("zfs_admin_snapshot=1"
> +"zfs_trim_extent_bytes_min=0")} will create the following file:
Sure, but it would be better to create a way to configure those module parameters
in Guix in a declarative way, first.
Your new kernel-loadable-module-service-type would be a good template
to write a kernel-module-configuration-service that can be extended by
other services. The latter should allow users to parametrize the kernel
in general via the operating-system form, and also allow services to extend
such configuration (with merging, and conflict detection).
It would be used:
(1) To declaratively specify the contents of something like /etc/modprobe.d .
It shouldn't even be called "/etc/modprobe.d"--it should also be in the store
instead. This directory is only useful for non-Linux-builtins.
(2) If the "module" is built-in then the kernel command line must get the
options instead. (in fact, it works as a kernel command line option also
if it's a loadable module--so not sure we need /etc/modprobe.d at all--at
least at first. But there's probably a maximal length for the kernel command
line that we could exceed if we did that long term)
I know it's annoying that Guix doesn't have this facility already, but the
time to introduce an interface for /etc/modprobe.d and the kernel command
line for builtin modules is before other services introduce their own
ad hoc way to create /etc/modprobe.d--like this tries to do here.
See also https://issues.guix.info/issue/42193 for an earlier attempt (which
is already very far--but it has a bug somewhere). There's also already a
kernel profile thing like you wrote in that patchset.
(Note that I would prefer there not to be a "LOAD?" in there because it
confuses loading the module (which is usually NOT started by user space
but by the kernel on its own) and confguring the module (which has to be
done by user space because it's specifying policy, not mechanism))
Also, because the kernel usually loads loadable modules on its own (potentially
really early), /etc/modprobe.d has to be preset and known to the modprobe
executable VERY EARLY (via environment variable MODPROBE_OPTIONS--see
gnu/services.scm %modprobe-wrapper).
It is totally possible that some modules in the initrd need options, too
(see load-linux-modules-from-directory for where this would need to go).
load-linux-module/fd already accepts options and flags--but both are not given
on the call. For this, part of future kernel-module-configuration entries
(the ones needed for modules in the initrd) should be copied into the initrd,
too.
Then there's the handoff between initrd and main system. It would be bad
if the kernel tried and succeeded to load a module that is not in the initrd
just before the modprobe.d directory is set up (because it would be loaded
without passing the options the user configured)--so that needs to be avoided.
@ludo: Could you help here?
> +@example
> +options zfs zfs_admin_snapshot=1 zfs_trim_extent_bytes_min=0
Note:
This can be usefully put in a modprobe.d-like directory if zfs is a module, but
not if it's built into the kernel. But it can be put into the kernel command
line in both cases.
But I guess the ZFS Linux kernel module can't be built-in into the kernel
anyway.
But that's a special case--in general, it's very much possible to make modules
built-in.
> + ;; You'd think we could've used kernel-module-loader-service-type,
Definitely.
> + ;; but the kernel-module-loader shepherd service is dependent on
> + ;; file-systems,
Yes--but why is that dependent on 'file-systems ? Is it because it needs /proc ?
Or is it an oversight ? I would prefer to get rid of this dependency and then
use kernel-module-loader-service-type.
Also, this manual loading of kernel modules is not supposed to be the way to
do things in Linux. That a kernel module was compiled as a module is
an *implementation detail*--so Linux should (and usually does) automatically
load kernel modules the first time a device for them is accessed (after all,
how would user space know whether something is compiled as a module or
built-in--that would be too much to ask).
Linux is not a microkernel, so the kind of modularily modprobe.d suggests
exists does not in fact exist in kernel space--even though Linux does a good
job faking it: modprobe.d contains:
* "alias": a feature to configure aliases, with wildcards (only one level of
aliases allowed!)
* "options" per module (also works for aliases with wildcards! That will be
"fun" to map to Guix)
* "install" in order to run some custom executable instead of loading the
module.
* "remove" in order to run some custom executable instead of unloading the
module.
* "blacklist" to ignore specific internal aliases of a module (that does
not do what one would intuitively think!).
If the file name of the regular file under /etc/modprobe.d is not used for
anything, then we can just have one file /gnu/store/*modprobe.d/guix.conf
in total in there.
Then there are sysctl kernel parameters--but those Guix already exposes via
sysctl-service-type. But those should also be made able to be extended
by other services, and merge conflicts should be handled. For example, users
often set net.ipv4.ip_forward=1 (for example via sysctl).
Thank you for all your effort to make ZFS work nicely in Guix.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45703] kernel-module-configuration-service for configuring kernel parameters
2021-01-06 19:41 ` [bug#45703] kernel-module-configuration-service for configuring kernel parameters Danny Milosavljevic
@ 2021-01-07 0:04 ` raid5atemyhomework via Guix-patches via
2021-01-07 5:38 ` [bug#45692] " raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-07 0:04 UTC (permalink / raw)
To: Danny Milosavljevic; +Cc: 45703, 45692
Hello Danny,
> See also https://issues.guix.info/issue/42193 for an earlier attempt (which
> is already very far--but it has a bug somewhere). There's also already a
> kernel profile thing like you wrote in that patchset.
> (Note that I would prefer there not to be a "LOAD?" in there because it
> confuses loading the module (which is usually NOT started by user space
> but by the kernel on its own) and confguring the module (which has to be
> done by user space because it's specifying policy, not mechanism))
Looks like that patchset was merged in, so basically I can just depend on that? So the first patch in this patchset would be dropped?
> But I guess the ZFS Linux kernel module can't be built-in into the kernel
> anyway.
>
> But that's a special case--in general, it's very much possible to make modules
> built-in.
ZFS *can* be built-in to the kernel, Ubuntu does it. You can't distribute it like that (Ubuntu distributes it like that but presumably they have enough lawyers to muddy the waters so that they can get away with it), but as the documentation in this patch notes: the user has every right to do whatever they want on the machine they own, including build a Linux kernel that has ZFS built-in and run it, they just can't make that version available to somebody else.
So to go whole-hog, we would have a service that replaces the kernel package and inserts kernel module sources in-tree somehow, then compiles Linux-libre on the user's machine. That would probably be a lot more painful to install ZFS with (the user has to recompile the whole kernel at each update of either kernel or ZFS, whereas with a kernel module the user has to recompile just the kernel module), so maybe kernel module is still better overall.
> > - ;; You'd think we could've used kernel-module-loader-service-type,
> >
> >
>
> Definitely.
>
> > - ;; but the kernel-module-loader shepherd service is dependent on
> >
> >
> > - ;; file-systems,
> >
> >
>
> Yes--but why is that dependent on 'file-systems ? Is it because it needs /proc ?
> Or is it an oversight ? I would prefer to get rid of this dependency and then
> use kernel-module-loader-service-type.
Dunno --- one VM I tested, I removed the `zfs-scan-automount` shepherd service from the `file-systems` target, and the VM still wouldn't boot, claiming a stack overflow (the same error which I got when I was still trying to use kernel-module-loader-service-type here). Or maybe I just got confused with which VM was which, testing VMs wasn't a stress-free vacation. I just want ZFS, because MD RAID5 ate my homework, this is getting tiresome...
One thing I notice about `kernel-module-loader-service-type` is that it's not instantiated in essential services, or indeed anywhere in Guix. A few services *do* extend it. But my **very rough** understanding is that if you're going to extend a service, it had better be instantiated *once* in the list of services.
In particular I note that the documentation for `kernel-module-loader-service-type` shows an example where it uses `service` to program the `kernel-module-loader-service-type`, not `simple-service`. This suggests to me that `kernel-module-loader-service-type` is broken because it's not in the list of essential services but is extensible. Maybe. It's designed as an extensible service, but isn't instantiated at default. Maybe that's what really bit me and not the shepherd circular dependency loop? *shrug*
>
> Also, this manual loading of kernel modules is not supposed to be the way to
> do things in Linux. That a kernel module was compiled as a module is
> an implementation detail--so Linux should (and usually does) automatically
> load kernel modules the first time a device for them is accessed (after all,
> how would user space know whether something is compiled as a module or
> built-in--that would be too much to ask).
So how do I get ZFS loaded? Note that the devices it targets are block devices and it needs to scan for block devices that are formatted for ZFS. Do other filesystems have some autoload rule?
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] kernel-module-configuration-service for configuring kernel parameters
2021-01-07 0:04 ` raid5atemyhomework via Guix-patches via
@ 2021-01-07 5:38 ` raid5atemyhomework via Guix-patches via
2021-01-07 9:16 ` [bug#42193] " raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-07 5:38 UTC (permalink / raw)
To: Danny Milosavljevic; +Cc: 45692
Hi Danny,
> > See also https://issues.guix.info/issue/42193 for an earlier attempt (which
> > is already very far--but it has a bug somewhere). There's also already a
> > kernel profile thing like you wrote in that patchset.
> > (Note that I would prefer there not to be a "LOAD?" in there because it
> > confuses loading the module (which is usually NOT started by user space
> > but by the kernel on its own) and confguring the module (which has to be
> > done by user space because it's specifying policy, not mechanism))
>
> Looks like that patchset was merged in, so basically I can just depend on that? So the first patch in this patchset would be dropped?
No, sorry, my mistake, only one patch completely unrelated to the actual new service type was merged in.
In any case --- is this objection something that would block this patchset from being added to Guix? I appreciate that we should "do it right" --- but there's also an argument for "keep it simple", and the first patch in this patchset gets us a good part of the way to what is needed.
It seems that my patch is equivalent to the existing WIP 2/6 `kernel-profile-service-type`, so maybe I can just steal that patch for now, write tests specific for it, and get on with running ZFS in production?
Can we at least get patch 3/4 of *this* patchset merged since it's a trivial bugfix? How about patch 2/4, which is not so trivial, but does give some flexibility in case other filesystems want to be as ambitious as ZFS is?
I imagine a later, much more comprehensive `kernel-module-configuration-service-type` can be built on top of `kernel-profile-service-type`, `kernel-module-loader-service-type`, and a later `kernel-module-options-service-type`. Then `zfs-service-type` can be modified to use that `kernel-module-configuration-service-type` instead of rolling its own bits and pieces.
----
On the other hand, if we want to think of `/` on ZFS, then we need a notion of kernel modules that are added to the `initrd` file, rather than to the `kernel` directory.
Diving into `gnu/system/linux-initrd.scm`, I note that we can provide a kernel package and a list of modules to copy from the kernel package to the `initrd` by a `(flat-linux-module-directory linux modules)` function.
I imagine that it would be possible to modify this as follows:
* Have `flat-linux-module-directory` accept a list of packages from which to find module names, not just a single package.
* Remove more code from `(operating-system-directory-base-entries os)`, and put the creation of the `"initrd"` file into a service, in much the same way that my patch does (unless there's an existing more-guixy way of putting files into `initrd`?).
* Create a service type that gathers packages whose modules are to be added to the `initrd`, and if that list is non-empty, pass it to a new key `#:extra-linux-module-packages` of the `initrd` function.
* `raw-initrd` would then pass that field together with the `linux` argument to the modified `flat-linux-module-directory` procedure.
* Create another service type that gathers module names to be loaded at `boot` and appends them to the `#:linux-modules` to the `initrd` function.
* Modify the `kernel-module-options-service-type` to pass in options via kernel command line always, maybe.
Then ZFS module can get loaded early, at boot, before the switch from `initrd` root to the "real" root.
* Create a service type that gathers additional pre-mount actions, which `raw-initrd` will accept as a new key `#:additional-pre-mount` and append to the `pre-mount` it passes to `boot-system`.
* The `zfs-service-type` would then extend this service type to pass in an action to perform a `zpool import -a -N`, which makes ZFS scan for devices containing ZFS pools.
* Somehow figure out a static build for the ZFS package, so we can use a statically-linked `zpool` in the above.
* The `zfs-service-type` already inherits a ZFS package from the given `base-package`, I imagine it would be possible to make further inheritance which modifies the build to be static. It does require an additional build, though. Maybe an additional `root-on-zfs?` field in `zfs-configuration` can gate this, so that we don't add the static build and extend the boot script if `(not root-on-zfs?)` (i.e. use `kernel-module-configuration-service-type` if `(not root-on-zfs?)`, else use the new put-it-in-the-initrd service types).
* Modify the various checks elsewhere so that ZFS poolnames can be passed as the `device` field of `file-system` records. See https://issues.guix.gnu.org/45643#3
* Then the installer can be modified so that if ZFS is included with the installer's profile, it can look at ZFS pools and offer to install to a ZFS pool instead of a `/dev` partition directly, then add `(file-system (device "rootpool") (mountpoint "/") (type "zfs"))` and the appropriate `zfs-service-type` and etc. and now we get `/` on ZFS.
But that can come later, much much later, I just need ZFS, *any* ZFS, before RAID5 eats more of my homework.
Thanks
raid5atemyhomework
> > But I guess the ZFS Linux kernel module can't be built-in into the kernel
> > anyway.
> > But that's a special case--in general, it's very much possible to make modules
> > built-in.
>
> ZFScan be built-in to the kernel, Ubuntu does it. You can't distribute it like that (Ubuntu distributes it like that but presumably they have enough lawyers to muddy the waters so that they can get away with it), but as the documentation in this patch notes: the user has every right to do whatever they want on the machine they own, including build a Linux kernel that has ZFS built-in and run it, they just can't make that version available to somebody else.
>
> So to go whole-hog, we would have a service that replaces the kernel package and inserts kernel module sources in-tree somehow, then compiles Linux-libre on the user's machine. That would probably be a lot more painful to install ZFS with (the user has to recompile the whole kernel at each update of either kernel or ZFS, whereas with a kernel module the user has to recompile just the kernel module), so maybe kernel module is still better overall.
>
> > > - ;; You'd think we could've used kernel-module-loader-service-type,
> > >
> > >
> >
> > Definitely.
> >
> > > - ;; but the kernel-module-loader shepherd service is dependent on
> > >
> > >
> > > - ;; file-systems,
> > >
> > >
> >
> > Yes--but why is that dependent on 'file-systems ? Is it because it needs /proc ?
> > Or is it an oversight ? I would prefer to get rid of this dependency and then
> > use kernel-module-loader-service-type.
>
> Dunno --- one VM I tested, I removed the`zfs-scan-automount` shepherd service from the `file-systems` target, and the VM still wouldn't boot, claiming a stack overflow (the same error which I got when I was still trying to use kernel-module-loader-service-type here). Or maybe I just got confused with which VM was which, testing VMs wasn't a stress-free vacation. I just want ZFS, because MD RAID5 ate my homework, this is getting tiresome...
>
> One thing I notice about `kernel-module-loader-service-type` is that it's not instantiated in essential services, or indeed anywhere in Guix. A few services do extend it. But my very rough understanding is that if you're going to extend a service, it had better be instantiated once in the list of services.
>
> In particular I note that the documentation for `kernel-module-loader-service-type` shows an example where it uses `service` to program the `kernel-module-loader-service-type`, not `simple-service`. This suggests to me that `kernel-module-loader-service-type` is broken because it's not in the list of essential services but is extensible. Maybe. It's designed as an extensible service, but isn't instantiated at default. Maybe that's what really bit me and not the shepherd circular dependency loop? shrug
>
> > Also, this manual loading of kernel modules is not supposed to be the way to
> > do things in Linux. That a kernel module was compiled as a module is
> > an implementation detail--so Linux should (and usually does) automatically
> > load kernel modules the first time a device for them is accessed (after all,
> > how would user space know whether something is compiled as a module or
> > built-in--that would be too much to ask).
>
> So how do I get ZFS loaded? Note that the devices it targets are block devices and it needs to scan for block devices that are formatted for ZFS. Do other filesystems have some autoload rule?
>
> Thanks
> raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 3/4] gnu: Fix ZFS package.
2021-01-06 15:56 ` [bug#45692] [PATCH 3/4] gnu: Fix ZFS package raid5atemyhomework via Guix-patches via
@ 2021-01-07 8:23 ` Danny Milosavljevic
0 siblings, 0 replies; 81+ messages in thread
From: Danny Milosavljevic @ 2021-01-07 8:23 UTC (permalink / raw)
Cc: raid5atemyhomework, 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 116 bytes --]
Pushed this one patch that fixes package "zfs" to guix master as
commit 8d3184c5bcf1a590f59a701f0484ab5cec673188.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#42193] kernel-module-configuration-service for configuring kernel parameters
2021-01-07 5:38 ` [bug#45692] " raid5atemyhomework via Guix-patches via
@ 2021-01-07 9:16 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-07 9:16 UTC (permalink / raw)
To: 45592@debbugs.gnu.org, 42193@debbugs.gnu.org; +Cc: Brice Waegeneire
Hi Danny and Brice,
Cross-posting to 42193 as well.
> > > See also https://issues.guix.info/issue/42193 for an earlier attempt (which
> > > is already very far--but it has a bug somewhere).
I think what 42193 WIP 2/6 is missing is this hunk from my own 45592 PATCH 1/4:
```diff
diff --git a/gnu/system.scm b/gnu/system.scm
index c284a18379..5c530f176e 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm@@ -625,26 +616,10 @@ possible (that is if there's a LINUX keyword argument in the build system)."
"Return the basic entries of the 'system' directory of OS for use as the
value of the SYSTEM-SERVICE-TYPE service."
(let* ((locale (operating-system-locale-directory os))
- (kernel (operating-system-kernel os))
(hurd (operating-system-hurd os))
- (modules (operating-system-kernel-loadable-modules os))
- (kernel (if hurd
- kernel
- (profile
- (content (packages->manifest
- (cons kernel
- (map (lambda (module)
- (if (package? module)
- (package-for-kernel kernel
- module)
- module))
- modules))))
- (hooks (list linux-module-database)))))
(initrd (and (not hurd) (operating-system-initrd-file os)))
(params (operating-system-boot-parameters-file os)))
- `(("kernel" ,kernel)
- ,@(if hurd `(("hurd" ,hurd)) '())
- ("parameters" ,params)
+ `(("parameters" ,params)
,@(if initrd `(("initrd" ,initrd)) '())
("locale" ,locale)))) ;used by libc
```
I hope we can progress this towards a workable ZFS-on-Guix.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 4/4] gnu: Add ZFS service.
2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
2021-01-06 19:41 ` [bug#45703] kernel-module-configuration-service for configuring kernel parameters Danny Milosavljevic
@ 2021-01-08 15:02 ` raid5atemyhomework via Guix-patches via
2021-01-09 8:31 ` raid5atemyhomework via Guix-patches via
2021-02-10 14:27 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
1 sibling, 2 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-08 15:02 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
This new version is dependent on https://issues.guix.info/45723 and https://issues.guix.info/45722
New features:
* We can now delay pool importation after particular mapped devices are opened or file systems are mounted. This is useful if you want to make a ZFS pool out of LUKS containers, or mount an encrypted ZFS dataset with the encryption key in another (hopefully also encrypted) filesystem.
* We can now put other non-ZFS filesystems inside ZVOLs, block devices provided by a ZFS pool, and have them mounted after ZFS has scanned pools and opened such ZVOLs for use by the ystem.
It's now possible to have a bunch of LUKS containers in a ZPOOL that provides a ZVOL that contains an EXT4 filesystem (though why would you do that, just use builtin ZFS encryption).
This uses my [PATCH 1/4] instead of https://issues.guix.info/42193#4. Should I use the latter instead? What would get ZFS on Guix faster?
I decided to remove ZFS parameter options, because I don't know whether I should wait for https://issues.guix.info/42193 and how long that would take.
From d0e095136cd471f083d92416f12ca22b47301f31 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 09:24:20 +0800
Subject: [PATCH 4/4] gnu: Add service to install ZFS.
* gnu/services/file-systems.scm: New file.
(zfs-service-type): New variable.
(<zfs-configuration>): New type.
(%zfs-zvol-dependency): New variable.
* gnu/local.mk: Add gnu/services/file-systems.scm.
* gnu/services/base.scm (dependency->shepherd-service-name): Export.
* doc/guix.texi (ZFS file system): New subsection.
---
doc/guix.texi | 107 +++++++++++++++++++++
gnu/local.mk | 1 +
gnu/services/base.scm | 4 +-
gnu/services/file-systems.scm | 174 ++++++++++++++++++++++++++++++++++
4 files changed, 285 insertions(+), 1 deletion(-)
create mode 100644 gnu/services/file-systems.scm
diff --git a/doc/guix.texi b/doc/guix.texi
index a31d355780..714274b8c9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13886,6 +13886,113 @@ a file system declaration such as:
compress-force=zstd,space_cache=v2"))
@end lisp
+@node ZFS file system
+@subsection ZFS file system
+
+The ZFS file system has a license incompatible with the Linux kernel,
+and thus cannot be distributed with the kernel. However, as a user
+you have the right to do whatever you want on your own hardware,
+including download the ZFS source code, compile it, link it to your
+own private copy of Linux, and run it. You can even use the Guix build
+system to automate this.
+
+As a large and complex kernel module, ZFS on Linux has to be compiled
+with a specific version of the kernel. Often even the latest ZFS
+package available cannot be compiled with the latest Linux kernel
+package provided by Guix System. Thus, installing the @code{zfs}
+package is likely to fail.
+
+Instead, you have to use an older long-term-support Linux kernel.
+Do not use @code{linux-libre-lts} as the latest long-term-support
+kernel might be too new for the ZFS package; instead, explicitly
+select the version number, like @code{linux-libre-5.4}.
+
+Then, you have to modify your system configuration file and use the
+selected older kernel, and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+ ; @dots{}
+ linux)
+(use-service-modules
+ ; @dots{}
+ file-systems)
+
+(define my-kernel linux-libre-5.4)
+
+(operating-system
+ (kernel my-kernel)
+ ;; @dots{}
+ (services
+ (cons* (service zfs-service-type
+ (zfs-configuration
+ (kernel my-kernel)))
+ ; @dots{}
+ %desktop-services))
+ ;; @dots{}
+ )
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type of the service to compile and install OpenZFS to
+your operating system. It loads the ZFS module at startup, imports
+pools, mounts automounted ZFS datasets, installs the ZFS command
+line tools, and can provide module options for the ZFS module. Its
+value must be a @code{zfs-configuration} record (see below).
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+ (zfs-configuration
+ (kernel linux-libre-5.4)
+ (options '("zfs_arc_max=5000000000"))))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS service.
+The available fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux kernel to compile ZFS for. Required.
+
+@item @code{base-zfs} (default: @code{zfs})
+The ZFS package to use. It will be modified to use the indicated
+kernel.
+
+@item @code{options} (default: @code{'()})
+A list of string options to pass as options to the ZFS module.
+These will be put in a @file{/etc/modprobe.d/zfs.conf} file,
+for example setting this to @code{'("zfs_admin_snapshot=1"
+"zfs_trim_extent_bytes_min=0")} will create the following file:
+
+@example
+options zfs zfs_admin_snapshot=1 zfs_trim_extent_bytes_min=0
+@end example
+@end table
+@end deftp
+
+Once your system has been configured to include the ZFS service
+and you have restarted the system, you can manage ZFS pools and
+datasets with @code{zpool} and @code{zfs} commands.
+
+ZFS datasets with an appropriate @code{mountpoint} property will
+be automounted at startup after the root file system is started.
+Encrypted datasets that are automounted will cause boot to pause
+and prompt for the password to be provided on the console.
+
+It's possible to have a ZFS dataset as your @code{/home} by simply
+setting the @code{mountpoint} property. However, note that ZFS will
+refuse to mount over a non-empty directory, so if your root
+filesystem already has a non-empty @code{/home}, you should remove
+it and its contents, then restart the system.
+
+Having ZFS as a root filesystem or as @code{/boot} is not supported
+yet.
+
@node Mapped Devices
@section Mapped Devices
diff --git a/gnu/local.mk b/gnu/local.mk
index 1151d4642e..5aeb45c4c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -602,6 +602,7 @@ GNU_SYSTEM_MODULES = \
%D%/services/dict.scm \
%D%/services/dns.scm \
%D%/services/docker.scm \
+ %D%/services/file-systems.scm \
%D%/services/authentication.scm \
%D%/services/games.scm \
%D%/services/ganeti.scm \
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 13cfb6a8a2..ef3680583b 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -188,7 +188,9 @@
references-file
- %base-services))
+ %base-services
+
+ dependency->shepherd-service-name))
;;; Commentary:
;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..9061ab9582
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,174 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+ #:use-module (gnu packages file-systems)
+ #:use-module (gnu services)
+ #:use-module (gnu services base)
+ #:use-module (gnu services linux)
+ #:use-module (gnu services shepherd)
+ #:use-module (gnu system mapped-devices)
+ #:use-module (guix gexp)
+ #:use-module (guix packages)
+ #:use-module (guix records)
+ #:export (zfs-service-type
+
+ zfs-configuration
+ zfs-configuration?
+ zfs-configuration-kernel
+ zfs-configuration-base-zfs
+ zfs-configuration-dependencies
+
+ %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+ zfs-configuration make-zfs-configuration zfs-configuration?
+ ; kernel you want to compile the base-zfs module for.
+ (kernel zfs-configuration-kernel)
+ ; base package that will be compiled for the kernel
+ (base-zfs zfs-configuration-base-zfs
+ (default zfs))
+ ; list of <mapped-device> | <file-system> we should wait for,
+ ; before scanning for ZFS pools.
+ (dependencies zfs-configuration-dependencies
+ (default '())))
+
+;; This is a synthetic and unusable MAPPED-DEVICE; its only use
+;; is to be added as a (dependency ...) of some FILE-SYSTEM.
+(define %zfs-zvol-dependency
+ (mapped-device
+ (source '())
+ ;; The /* prevents naming conflict with non-ZFS device mappings,
+ ;; since it is not a valid name for mapped devices, and also
+ ;; implies "all zvols" in terms of globs.
+ (targets '("zvol/*"))
+ (type #f)))
+
+(define (make-zfs-package conf)
+ (let ((base-zfs (zfs-configuration-base-zfs conf))
+ (kernel (zfs-configuration-kernel conf)))
+ (package
+ (inherit base-zfs)
+ (name (string-join (list (package-name base-zfs)
+ "for"
+ (package-name kernel)
+ (package-version kernel)
+ "version")
+ "-"))
+ (arguments (cons* #:linux kernel (package-arguments base-zfs))))))
+
+(define (zfs-loadable-module conf)
+ (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (zfs (file-append zfs-package "/sbin/zfs"))
+ (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
+ (scheme-modules `((srfi srfi-1)
+ (srfi srfi-34)
+ (srfi srfi-35)
+ (rnrs io ports)
+ ,@%default-modules)))
+
+ (define zfs-scan
+ (shepherd-service
+ (provision '(zfs-scan))
+ (documentation "Scans for ZFS pools.")
+ (requirement `(kernel-module-loader
+ root-file-system
+ ,@(map dependency->shepherd-service-name
+ (zfs-configuration-dependencies conf))))
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "error importing zpools: ~a~%"
+ (condition-message c))
+ #f))
+ ; TODO: optionally use a cachefile, for systems with dozens or
+ ; hundreds of devices.
+ (invoke/quiet #$zpool "import" "-a" "-N"))))
+ (stop #~(const #t))))
+
+ (define device-mapping-zvol/*
+ (shepherd-service
+ (provision '(device-mapping-zvol/*))
+ (documentation "Waits for ZFS ZVOL devices to appear.")
+ (requirement '(zfs-scan))
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "error waiting for zvols: ~a~%"
+ (condition-message c))
+ #f))
+ (invoke/quiet #$zvol_wait))))
+ (stop #~(const #t))))
+
+ (define zfs-automount
+ (shepherd-service
+ (provision '(zfs-automount))
+ (documentation "Automounts ZFS datasets.")
+ (requirement '(zfs-scan))
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "error automounting zfs: ~a~$")
+ #f))
+ ; (current-output-port) is typically connected to /dev/klog,
+ ; so redirect it to (current-error-port) so that user can see
+ ; prompts for passphrases on console
+ (with-output-to-port (current-error-port)
+ (lambda ()
+ (invoke #$zfs "mount" "-a" "-l"))))))
+ (stop #~(lambda _
+ ;; make sure we don't keep any ZFS mountpoints busy.
+ (chdir "/")
+ ;; unmount everything.
+ (invoke/quiet #$zfs "unmount" "-a" "-f")))))
+
+ (list zfs-scan
+ device-mapping-zvol/*
+ zfs-automount)))
+
+(define zfs-service-type
+ (service-type (name 'zfs)
+ (extensions
+ (list
+ ; install the kernel module
+ (service-extension kernel-loadable-module-service-type
+ zfs-loadable-module)
+ ; load the kernel module
+ (service-extension kernel-module-loader-service-type
+ (const '("zfs")))
+ ; scan ZFS pools, automount filesystem, wait for zvols.
+ (service-extension shepherd-root-service-type
+ zfs-shepherd-services)
+ ; make sure automount occurs before file-systems target is reached
+ (service-extension file-systems-target-service-type
+ (const '(zfs-automount)))
+ ; install ZFS management tools
+ (service-extension profile-service-type
+ (compose list make-zfs-package))
+ ; install ZFS udev rules
+ (service-extension udev-service-type
+ (compose list make-zfs-package))))
+ (description
+ "Install ZFS, an advanced filesystem and volume manager.")))
--
2.30.0
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules.
2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
@ 2021-01-08 16:16 ` raid5atemyhomework via Guix-patches via
2021-02-10 14:13 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
2021-02-10 15:44 ` Ludovic Courtès
2 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-08 16:16 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
Is this patch acceptable? https://lists.nongnu.org/archive/html/guix-devel/2021-01/msg00070.html I reported that I compared a `guix system build` result of an `operating-system` that used the existing `(kernel-loadable-modules ...)` field, with and without this patch.
The resulting builds resulted in different hashes, but with exactly the same contents in the build --- `diff -r` very quickly reported no differences because it saw that nearly all the symlinks pointed to the same gnu store items.
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 4/4] gnu: Add ZFS service.
2021-01-08 15:02 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
@ 2021-01-09 8:31 ` raid5atemyhomework via Guix-patches via
2021-02-08 3:31 ` Danny Milosavljevic
2021-02-10 14:27 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
1 sibling, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-01-09 8:31 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
New version again. Change: Fix documentation `doc/guix.texi` to remove mentions of `options` and add documentation about the new `dependencies` features.
From dfe9ad7512d348933beb5b42041965ff604422b5 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 09:24:20 +0800
Subject: [PATCH 1/2] gnu: Add service to install ZFS.
* gnu/services/file-systems.scm: New file.
(zfs-service-type): New variable.
(<zfs-configuration>): New type.
(%zfs-zvol-dependency): New variable.
* gnu/local.mk: Add gnu/services/file-systems.scm.
* gnu/services/base.scm (dependency->shepherd-service-name): Export.
* doc/guix.texi (ZFS file system): New subsection.
---
doc/guix.texi | 129 +++++++++++++++++++++++++
gnu/local.mk | 1 +
gnu/services/base.scm | 4 +-
gnu/services/file-systems.scm | 174 ++++++++++++++++++++++++++++++++++
4 files changed, 307 insertions(+), 1 deletion(-)
create mode 100644 gnu/services/file-systems.scm
diff --git a/doc/guix.texi b/doc/guix.texi
index a31d355780..7004efe3c4 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13886,6 +13886,135 @@ a file system declaration such as:
compress-force=zstd,space_cache=v2"))
@end lisp
+@node ZFS file system
+@subsection ZFS file system
+
+The ZFS file system has a license incompatible with the Linux kernel,
+and thus cannot be distributed with the kernel. However, as a user
+you have the right to do whatever you want on your own hardware,
+including download the ZFS source code, compile it, link it to your
+own private copy of Linux, and run it. You can even use the Guix build
+system to automate this.
+
+As a large and complex kernel module, ZFS on Linux has to be compiled
+with a specific version of the kernel. Often even the latest ZFS
+package available cannot be compiled with the latest Linux kernel
+package provided by Guix System. Thus, installing the @code{zfs}
+package is likely to fail.
+
+Instead, you have to use an older long-term-support Linux kernel.
+Do not use @code{linux-libre-lts}, since the latest long-term-support
+kernel might be too new for the ZFS package; instead, explicitly
+select the version number, like @code{linux-libre-5.4}, and upgrade
+it manually later when you have verified that the ZFS version
+available on Guix can be compiled with a later LTS kernel.
+
+Then, you have to modify your system configuration file and use the
+selected older kernel, and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+ ; @dots{}
+ linux)
+(use-service-modules
+ ; @dots{}
+ file-systems)
+
+(define my-kernel linux-libre-5.4)
+
+(operating-system
+ (kernel my-kernel)
+ ;; @dots{}
+ (services
+ (cons* (service zfs-service-type
+ (zfs-configuration
+ (kernel my-kernel)))
+ ; @dots{}
+ %desktop-services))
+ ;; @dots{}
+ )
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type of the service to compile and install OpenZFS to
+your operating system. It loads the ZFS module at startup, imports
+pools, mounts automounted ZFS datasets, and installs the ZFS command
+line tools. Its value must be a @code{zfs-configuration} record
+(see below).
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+ (zfs-configuration
+ (kernel linux-libre-5.4)))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS service.
+The available fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux kernel to compile ZFS for. Required. It
+@emph{must} be the same kernel you use in your operating system.
+
+@item @code{base-zfs} (default: @code{zfs})
+The ZFS package to use. It will be modified to use the indicated
+kernel.
+
+@item @code{dependencies} (default: @code{'()})
+A list of @code{<file-system>}s or @code{<mapped-device>}s that
+must be mounted or opened before ZFS scans for pools to import.
+For example, you might have LUKS containers as the leaf VDEVs of
+a ZFS pool.
+@end table
+@end deftp
+
+Once your system has been configured to include the ZFS service
+and you have restarted the system, you can manage ZFS pools and
+datasets with @code{zpool} and @code{zfs} commands.
+
+ZFS datasets with an appropriate @code{mountpoint} property will
+be automounted at startup after the root file system is started.
+Encrypted datasets that are automounted will cause boot to pause
+and prompt for the password to be provided on the console.
+
+It's possible to have a ZFS dataset as your @code{/home} by simply
+setting the @code{mountpoint} ZFS property. However, note that ZFS
+will refuse to mount over a non-empty directory, so if your root
+filesystem already has a non-empty @code{/home}, you should remove
+it and its contents, then restart the system.
+
+ZFS features @dfn{ZVOL}s, which are block devices that ZFS exposes
+to the system. You can put any file system inside a ZVOL. In order
+to mount such a filesystem at boot, you can declare it as dependent
+on @code{%zfs-zvol-dependency}.
+
+@defvr {Scheme Variable} %zfs-zvol-dependency
+This is an artificial @code{<mapped-device>} object which tells
+the file mounting service to wait for ZFS to provide ZVOLs before
+mounting the file system that is dependent on it. For example:
+
+@lisp
+(file-system
+ (device "/dev/zvol/pool-name/zvol-name")
+ (mount-point "/ext4-on-zfs")
+ (type "ext4")
+ (dependencies (list %zfs-zvol-dependency)))
+@end lisp
+
+Do @emph{not} add @code{%zfs-zvol-dependency} to your
+@code{mapped-devices} declaration, In addition, only use it as a
+@code{file-system} dependency if you instantiate a
+@code{zfs-service-type} service in your operating system.
+@end defvr
+
+Having ZFS as a root filesystem or as @code{/boot} is not supported
+yet.
+
@node Mapped Devices
@section Mapped Devices
diff --git a/gnu/local.mk b/gnu/local.mk
index 1151d4642e..5aeb45c4c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -602,6 +602,7 @@ GNU_SYSTEM_MODULES = \
%D%/services/dict.scm \
%D%/services/dns.scm \
%D%/services/docker.scm \
+ %D%/services/file-systems.scm \
%D%/services/authentication.scm \
%D%/services/games.scm \
%D%/services/ganeti.scm \
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 13cfb6a8a2..ef3680583b 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -188,7 +188,9 @@
references-file
- %base-services))
+ %base-services
+
+ dependency->shepherd-service-name))
;;; Commentary:
;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..9061ab9582
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,174 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+ #:use-module (gnu packages file-systems)
+ #:use-module (gnu services)
+ #:use-module (gnu services base)
+ #:use-module (gnu services linux)
+ #:use-module (gnu services shepherd)
+ #:use-module (gnu system mapped-devices)
+ #:use-module (guix gexp)
+ #:use-module (guix packages)
+ #:use-module (guix records)
+ #:export (zfs-service-type
+
+ zfs-configuration
+ zfs-configuration?
+ zfs-configuration-kernel
+ zfs-configuration-base-zfs
+ zfs-configuration-dependencies
+
+ %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+ zfs-configuration make-zfs-configuration zfs-configuration?
+ ; kernel you want to compile the base-zfs module for.
+ (kernel zfs-configuration-kernel)
+ ; base package that will be compiled for the kernel
+ (base-zfs zfs-configuration-base-zfs
+ (default zfs))
+ ; list of <mapped-device> | <file-system> we should wait for,
+ ; before scanning for ZFS pools.
+ (dependencies zfs-configuration-dependencies
+ (default '())))
+
+;; This is a synthetic and unusable MAPPED-DEVICE; its only use
+;; is to be added as a (dependency ...) of some FILE-SYSTEM.
+(define %zfs-zvol-dependency
+ (mapped-device
+ (source '())
+ ;; The /* prevents naming conflict with non-ZFS device mappings,
+ ;; since it is not a valid name for mapped devices, and also
+ ;; implies "all zvols" in terms of globs.
+ (targets '("zvol/*"))
+ (type #f)))
+
+(define (make-zfs-package conf)
+ (let ((base-zfs (zfs-configuration-base-zfs conf))
+ (kernel (zfs-configuration-kernel conf)))
+ (package
+ (inherit base-zfs)
+ (name (string-join (list (package-name base-zfs)
+ "for"
+ (package-name kernel)
+ (package-version kernel)
+ "version")
+ "-"))
+ (arguments (cons* #:linux kernel (package-arguments base-zfs))))))
+
+(define (zfs-loadable-module conf)
+ (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (zfs (file-append zfs-package "/sbin/zfs"))
+ (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
+ (scheme-modules `((srfi srfi-1)
+ (srfi srfi-34)
+ (srfi srfi-35)
+ (rnrs io ports)
+ ,@%default-modules)))
+
+ (define zfs-scan
+ (shepherd-service
+ (provision '(zfs-scan))
+ (documentation "Scans for ZFS pools.")
+ (requirement `(kernel-module-loader
+ root-file-system
+ ,@(map dependency->shepherd-service-name
+ (zfs-configuration-dependencies conf))))
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "error importing zpools: ~a~%"
+ (condition-message c))
+ #f))
+ ; TODO: optionally use a cachefile, for systems with dozens or
+ ; hundreds of devices.
+ (invoke/quiet #$zpool "import" "-a" "-N"))))
+ (stop #~(const #t))))
+
+ (define device-mapping-zvol/*
+ (shepherd-service
+ (provision '(device-mapping-zvol/*))
+ (documentation "Waits for ZFS ZVOL devices to appear.")
+ (requirement '(zfs-scan))
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "error waiting for zvols: ~a~%"
+ (condition-message c))
+ #f))
+ (invoke/quiet #$zvol_wait))))
+ (stop #~(const #t))))
+
+ (define zfs-automount
+ (shepherd-service
+ (provision '(zfs-automount))
+ (documentation "Automounts ZFS datasets.")
+ (requirement '(zfs-scan))
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "error automounting zfs: ~a~$")
+ #f))
+ ; (current-output-port) is typically connected to /dev/klog,
+ ; so redirect it to (current-error-port) so that user can see
+ ; prompts for passphrases on console
+ (with-output-to-port (current-error-port)
+ (lambda ()
+ (invoke #$zfs "mount" "-a" "-l"))))))
+ (stop #~(lambda _
+ ;; make sure we don't keep any ZFS mountpoints busy.
+ (chdir "/")
+ ;; unmount everything.
+ (invoke/quiet #$zfs "unmount" "-a" "-f")))))
+
+ (list zfs-scan
+ device-mapping-zvol/*
+ zfs-automount)))
+
+(define zfs-service-type
+ (service-type (name 'zfs)
+ (extensions
+ (list
+ ; install the kernel module
+ (service-extension kernel-loadable-module-service-type
+ zfs-loadable-module)
+ ; load the kernel module
+ (service-extension kernel-module-loader-service-type
+ (const '("zfs")))
+ ; scan ZFS pools, automount filesystem, wait for zvols.
+ (service-extension shepherd-root-service-type
+ zfs-shepherd-services)
+ ; make sure automount occurs before file-systems target is reached
+ (service-extension file-systems-target-service-type
+ (const '(zfs-automount)))
+ ; install ZFS management tools
+ (service-extension profile-service-type
+ (compose list make-zfs-package))
+ ; install ZFS udev rules
+ (service-extension udev-service-type
+ (compose list make-zfs-package))))
+ (description
+ "Install ZFS, an advanced filesystem and volume manager.")))
--
2.30.0
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services.
2021-01-06 15:55 ` [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services raid5atemyhomework via Guix-patches via
@ 2021-01-23 13:05 ` 宋文武
2021-01-25 0:18 ` guix-patches--- via
0 siblings, 1 reply; 81+ messages in thread
From: 宋文武 @ 2021-01-23 13:05 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 876 bytes --]
Hello!
raid5atemyhomework <raid5atemyhomework@protonmail.com> writes:
>>From 792a8f8efc95e4fe9a94d42f839ddcfb034b8540 Mon Sep 17 00:00:00 2001
> From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
> Date: Wed, 6 Jan 2021 08:15:54 +0800
> Subject: [PATCH 2/4] gnu: Make file-systems target extensible by services.
It’s not clear to me what “file-systems target” is, and why we’re
extending ‘file-systems-target-service-type’… I think what we want is
to extend the ‘file-system-service-type’ with shepherd services’ names,
which means some shepherd services that will handle file systems
mounting themself instead of the usual <file-system> objects, fstab
entries, mounted by kernel.
So I write with this patch to extend file-system-service-type directly
instead of introducing a new ‘file-systems-target-service-type’:
[-- Attachment #2: 0001-services-Allow-file-system-service-type-extensible-b.patch --]
[-- Type: text/x-patch, Size: 3365 bytes --]
From 44ee1e470a2f9d4985af4d51654d9f943caa0f24 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
Date: Sat, 23 Jan 2021 20:39:06 +0800
Subject: [PATCH] services: Allow 'file-system-service-type' extensible by
service name.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
* gnu/services/base.scm (file-system-shepherd-services): Add
'extra-services-names' paramater.
(file-system-service-type): Handle services’ names from extensions.
---
gnu/services/base.scm | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index f6a490f712..7bddef5034 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -364,15 +364,16 @@ FILE-SYSTEM."
(gnu system file-systems)
,@%default-modules)))))))
-(define (file-system-shepherd-services file-systems)
+(define (file-system-shepherd-services file-systems extra-services-names)
"Return the list of Shepherd services for FILE-SYSTEMS."
(let* ((file-systems (filter file-system-mount? file-systems)))
(define sink
(shepherd-service
(provision '(file-systems))
- (requirement (cons* 'root-file-system 'user-file-systems
- (map file-system->shepherd-service-name
- file-systems)))
+ (requirement (append '(root-file-system user-file-systems)
+ (map file-system->shepherd-service-name
+ file-systems)
+ extra-services-names))
(documentation "Target for all the initially-mounted file systems")
(start #~(const #t))
(stop #~(const #f))))
@@ -429,13 +430,23 @@ FILE-SYSTEM."
(service-type (name 'file-systems)
(extensions
(list (service-extension shepherd-root-service-type
- file-system-shepherd-services)
+ (lambda (value)
+ (file-system-shepherd-services
+ (filter file-system? value)
+ (filter symbol? value))))
(service-extension fstab-service-type
- file-system-fstab-entries)
+ (lambda (value)
+ (file-system-fstab-entries
+ (filter file-system? value))))
;; Have 'user-processes' depend on 'file-systems'.
(service-extension user-processes-service-type
(const '(file-systems)))))
+
+ ;; Extensions consist of lists of <file-system> objects or
+ ;; shepherd services’ names (symbols). In the latter case,
+ ;; the provided shepherd services supposed to mount and
+ ;; unmount some file systems themself.
(compose concatenate)
(extend append)
(description
--
2.29.2
[-- Attachment #3: Type: text/plain, Size: 32 bytes --]
What do you think? Thank you!
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services.
2021-01-23 13:05 ` 宋文武
@ 2021-01-25 0:18 ` guix-patches--- via
2021-02-10 14:17 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
0 siblings, 1 reply; 81+ messages in thread
From: guix-patches--- via @ 2021-01-25 0:18 UTC (permalink / raw)
To: 宋文武; +Cc: 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]
Hello as well,
That is certainly another possibility, would this be more palatable to Guix?
Thanks
raid5atemyhomework
> Hello!
>
> raid5atemyhomework raid5atemyhomework@protonmail.com writes:
>
> > > From 792a8f8efc95e4fe9a94d42f839ddcfb034b8540 Mon Sep 17 00:00:00 2001
> > > From: raid5atemyhomework raid5atemyhomework@protonmail.com
> > > Date: Wed, 6 Jan 2021 08:15:54 +0800
> > > Subject: [PATCH 2/4] gnu: Make file-systems target extensible by services.
>
> It’s not clear to me what “file-systems target” is, and why we’re
> extending ‘file-systems-target-service-type’… I think what we want is
> to extend the ‘file-system-service-type’ with shepherd services’ names,
> which means some shepherd services that will handle file systems
> mounting themself instead of the usual <file-system> objects, fstab
> entries, mounted by kernel.
>
> So I write with this patch to extend file-system-service-type directly
> instead of introducing a new ‘file-systems-target-service-type’:
>
> What do you think? Thank you!
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-services-Allow-file-system-service-type-extensible-b.patch --]
[-- Type: text/x-patch; name=0001-services-Allow-file-system-service-type-extensible-b.patch, Size: 3295 bytes --]
From 44ee1e470a2f9d4985af4d51654d9f943caa0f24 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
Date: Sat, 23 Jan 2021 20:39:06 +0800
Subject: [PATCH] services: Allow 'file-system-service-type' extensible by
service name.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
* gnu/services/base.scm (file-system-shepherd-services): Add
'extra-services-names' paramater.
(file-system-service-type): Handle services’ names from extensions.
---
gnu/services/base.scm | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index f6a490f712..7bddef5034 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -364,15 +364,16 @@ FILE-SYSTEM."
(gnu system file-systems)
,@%default-modules)))))))
-(define (file-system-shepherd-services file-systems)
+(define (file-system-shepherd-services file-systems extra-services-names)
"Return the list of Shepherd services for FILE-SYSTEMS."
(let* ((file-systems (filter file-system-mount? file-systems)))
(define sink
(shepherd-service
(provision '(file-systems))
- (requirement (cons* 'root-file-system 'user-file-systems
- (map file-system->shepherd-service-name
- file-systems)))
+ (requirement (append '(root-file-system user-file-systems)
+ (map file-system->shepherd-service-name
+ file-systems)
+ extra-services-names))
(documentation "Target for all the initially-mounted file systems")
(start #~(const #t))
(stop #~(const #f))))
@@ -429,13 +430,23 @@ FILE-SYSTEM."
(service-type (name 'file-systems)
(extensions
(list (service-extension shepherd-root-service-type
- file-system-shepherd-services)
+ (lambda (value)
+ (file-system-shepherd-services
+ (filter file-system? value)
+ (filter symbol? value))))
(service-extension fstab-service-type
- file-system-fstab-entries)
+ (lambda (value)
+ (file-system-fstab-entries
+ (filter file-system? value))))
;; Have 'user-processes' depend on 'file-systems'.
(service-extension user-processes-service-type
(const '(file-systems)))))
+
+ ;; Extensions consist of lists of <file-system> objects or
+ ;; shepherd services’ names (symbols). In the latter case,
+ ;; the provided shepherd services supposed to mount and
+ ;; unmount some file systems themself.
(compose concatenate)
(extend append)
(description
--
2.29.2
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 4/4] gnu: Add ZFS service.
2021-01-09 8:31 ` raid5atemyhomework via Guix-patches via
@ 2021-02-08 3:31 ` Danny Milosavljevic
2021-02-08 6:25 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: Danny Milosavljevic @ 2021-02-08 3:31 UTC (permalink / raw)
To: 45692; +Cc: raid5atemyhomework
[-- Attachment #1: Type: text/plain, Size: 389 bytes --]
Hello,
is it necessary to manually load the ZFS Linux kernel module? Usually, Linux
should autoload its drivers when the device files are accessed--module or not.
If that fails, didn't you patch the zfs package so it refers to modprobe anyway?
(I wanna prevent having 543 places in Guix where modprobe is invoked)
If it doesn't autoload, we can totally modprobe it, though.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 4/4] gnu: Add ZFS service.
2021-02-08 3:31 ` Danny Milosavljevic
@ 2021-02-08 6:25 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-02-08 6:25 UTC (permalink / raw)
To: Danny Milosavljevic; +Cc: 45692
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, February 8, 2021 3:31 AM, Danny Milosavljevic <dannym@scratchpost.org> wrote:
> Hello,
>
> is it necessary to manually load the ZFS Linux kernel module? Usually, Linux
> should autoload its drivers when the device files are accessed--module or not.
>
> If that fails, didn't you patch the zfs package so it refers to modprobe anyway?
It does, for `udev`. Not certain if it would trigger the load, though. Presumably it would, but I have not tested.
A lot of this code was written in 0.8.5, where the ZFS tools will *not* attempt to `modprobe` the module first, they just error out. I think in 2.0.1 upstream added code to load the module if the ZFS tool(s) were invoked while the module was not loaded, which is why additional patch was added by me for the 2.0.1 to patch in `modprobe`. I have not tested if the `modprobe`-ing by the ZFS tools actually *works*, though.
Maybe when I find some amount of motivation I can go test it.
>
> (I wanna prevent having 543 places in Guix where modprobe is invoked)
>
> If it doesn't autoload, we can totally modprobe it, though.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
2021-01-08 16:16 ` raid5atemyhomework via Guix-patches via
@ 2021-02-10 14:13 ` Ludovic Courtès
2021-02-10 15:44 ` Ludovic Courtès
2 siblings, 0 replies; 81+ messages in thread
From: Ludovic Courtès @ 2021-02-10 14:13 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: Danny Milosavljevic, 45692@debbugs.gnu.org
raid5atemyhomework <raid5atemyhomework@protonmail.com> skribis:
> +(define (kernel-builder-configuration->system-entry config)
> + "Return the kernel and hurd entries of the 'system' directory."
> + (mbegin %store-monad
> + (let* ((kernel (kernel-builder-configuration-kernel config))
> + (hurd (kernel-builder-configuration-hurd config))
> + (modules (kernel-builder-configuration-modules config))
> + (kernel (if hurd
> + kernel
> + (profile
> + (content (packages->manifest
> + (cons kernel
> + (map (lambda (module)
> + (if (package? module)
> + (package-for-kernel kernel module)
> + module))
> + modules))))
> + (hooks (list linux-module-database))))))
> + (return `(("kernel" ,kernel)
> + ,@(if hurd `(("hurd" ,hurd)) '()))))))
It may be clearer to avoid ‘mbegin’ and instead write it this way:
(define (kernel-builder-configuration-modules config)
(let* …
(with-monad %store-monad
(return …))))
Both work but I find this variant slightly clearer.
Ludo’.
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
2021-01-25 0:18 ` guix-patches--- via
@ 2021-02-10 14:17 ` Ludovic Courtès
2021-02-10 14:46 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: Ludovic Courtès @ 2021-02-10 14:17 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 宋文武, 45692@debbugs.gnu.org
Hi!
I agree with 宋文武.
Still…
raid5atemyhomework <raid5atemyhomework@protonmail.com> skribis:
> (service-type (name 'file-systems)
> (extensions
> (list (service-extension shepherd-root-service-type
> - file-system-shepherd-services)
> + (lambda (value)
> + (file-system-shepherd-services
> + (filter file-system? value)
> + (filter symbol? value))))
> (service-extension fstab-service-type
> - file-system-fstab-entries)
> + (lambda (value)
> + (file-system-fstab-entries
> + (filter file-system? value))))
>
> ;; Have 'user-processes' depend on 'file-systems'.
> (service-extension user-processes-service-type
> (const '(file-systems)))))
> +
> + ;; Extensions consist of lists of <file-system> objects or
> + ;; shepherd services’ names (symbols). In the latter case,
> + ;; the provided shepherd services supposed to mount and
> + ;; unmount some file systems themself.
Why do we need to extend with symbols?
In general it’s much clearer if extensions receive only one type of
object (<file-system> records in this case). It’s also best to avoid
passing around symbolic names (that’s why we extend with <file-system>
records rather than with Shepherd service names or whatever.)
Ludo’.
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
2021-01-08 15:02 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
2021-01-09 8:31 ` raid5atemyhomework via Guix-patches via
@ 2021-02-10 14:27 ` Ludovic Courtès
2021-02-10 14:32 ` raid5atemyhomework via Guix-patches via
2021-02-13 1:49 ` raid5atemyhomework via Guix-patches via
1 sibling, 2 replies; 81+ messages in thread
From: Ludovic Courtès @ 2021-02-10 14:27 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 45692@debbugs.gnu.org
raid5atemyhomework <raid5atemyhomework@protonmail.com> skribis:
>>From d0e095136cd471f083d92416f12ca22b47301f31 Mon Sep 17 00:00:00 2001
> From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
> Date: Wed, 6 Jan 2021 09:24:20 +0800
> Subject: [PATCH 4/4] gnu: Add service to install ZFS.
>
> * gnu/services/file-systems.scm: New file.
> (zfs-service-type): New variable.
> (<zfs-configuration>): New type.
> (%zfs-zvol-dependency): New variable.
> * gnu/local.mk: Add gnu/services/file-systems.scm.
> * gnu/services/base.scm (dependency->shepherd-service-name): Export.
> * doc/guix.texi (ZFS file system): New subsection.
A bit of nitpicking on the documentation bits…
> +@node ZFS file system
> +@subsection ZFS file system
Please follow the same typographical conventions as in the rest of the
manual. In particular, capitalize section titles and leave two spaces
after end-of-sentence periods.
Here I recommend s/ZFS file system/ZFS/ in all the text.
> +The ZFS file system has a license incompatible with the Linux kernel,
> +and thus cannot be distributed with the kernel. However, as a user
> +you have the right to do whatever you want on your own hardware,
> +including download the ZFS source code, compile it, link it to your
> +own private copy of Linux, and run it. You can even use the Guix build
> +system to automate this.
How about:
ZFS is free software; unfortunately its license is incompatible with
the GNU General Public License (GPL), the license of the Linux kernel,
which means they cannot be distributed together. However, as a user,
you can choose to build ZFS and use it together with Linux; you can
even rely on Guix to automate this task. See
@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
the Free Software Foundation} for more information.
?
> +As a large and complex kernel module, ZFS on Linux has to be compiled
^
I think it’s a proper noun and is spelled “ZFS-on-Linux”.
> +with a specific version of the kernel. Often even the latest ZFS
> +package available cannot be compiled with the latest Linux kernel
^
s/Linux/Linux-libre/
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
2021-02-10 14:27 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
@ 2021-02-10 14:32 ` raid5atemyhomework via Guix-patches via
2021-02-13 1:49 ` raid5atemyhomework via Guix-patches via
1 sibling, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-02-10 14:32 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 45692@debbugs.gnu.org
Hi Ludo,
> raid5atemyhomework raid5atemyhomework@protonmail.com skribis:
>
> > > From d0e095136cd471f083d92416f12ca22b47301f31 Mon Sep 17 00:00:00 2001
> > > From: raid5atemyhomework raid5atemyhomework@protonmail.com
> > > Date: Wed, 6 Jan 2021 09:24:20 +0800
> > > Subject: [PATCH 4/4] gnu: Add service to install ZFS.
> >
> > - gnu/services/file-systems.scm: New file.
> > (zfs-service-type): New variable.
> > (<zfs-configuration>): New type.
> > (%zfs-zvol-dependency): New variable.
> >
> > - gnu/local.mk: Add gnu/services/file-systems.scm.
> > - gnu/services/base.scm (dependency->shepherd-service-name): Export.
> > - doc/guix.texi (ZFS file system): New subsection.
>
> A bit of nitpicking on the documentation bits…
>
> > +@node ZFS file system
> > +@subsection ZFS file system
>
> Please follow the same typographical conventions as in the rest of the
> manual. In particular, capitalize section titles and leave two spaces
> after end-of-sentence periods.
This was imitated from the `@subsection Btrfs file system`.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
2021-02-10 14:17 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
@ 2021-02-10 14:46 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-02-10 14:46 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 宋文武, 45692@debbugs.gnu.org
> Why do we need to extend with symbols?
>
> In general it’s much clearer if extensions receive only one type of
> object (<file-system> records in this case). It’s also best to avoid
> passing around symbolic names (that’s why we extend with <file-system>
> records rather than with Shepherd service names or whatever.)
For this case, how would it be done?
ZFS file system, on other operating systems and distributions, is documented as automatically mounting filesystems, without management in an `fstab` or similar file, because the intent is that you would make lots of filesystems for various uses and managing an `fstab` would be too onerous. Thus, ZFS file system expects to mount multiple file systems with a single `zfs mount -a` command at startup.
Would the below sketch be acceptable?
```scheme
; gnu/system/file-systems.scm
(define-record-type* file-system #;...
#;...
(has-fstab-entry? file-system-has-fstab-entry? (default #t)))
;...
; gnu/services/base,scm
(define file-system-service-type
(service-type
#;...
(extensions
(list #;...
(service-extension fstab-service-type
(lambda (file-systems)
(filter file-system-has-fstab-entry?
file-systems)))
#;...))
#;...))
;...
; gnu/services/file-systems.scm
(define zfs-service-type
(service-type
#;...
(extensions (list #;...
(service-extension file-system-service-type
(const (list (file-system
(device "dummy")
(mount-point "zfs/*")
(has-fstab-entry? #f)))))))
#;...))
```
Then there will be a Shepherd service providing `file-system-zfs/*` which would perform `zfs mount -a -l` on `start` and `zfs unmount -a -f` on `stop`.
Would that be acceptable? I am wary of this since it creates a dummy file-system and needs an additional field on every `file-system` record, one which is *only* used by ZFS. I feel the `file-system-target-service-type` is more generic and does not use trickery.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
2021-01-08 16:16 ` raid5atemyhomework via Guix-patches via
2021-02-10 14:13 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
@ 2021-02-10 15:44 ` Ludovic Courtès
2021-02-10 16:49 ` raid5atemyhomework via Guix-patches via
2 siblings, 1 reply; 81+ messages in thread
From: Ludovic Courtès @ 2021-02-10 15:44 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: Danny Milosavljevic, 45692@debbugs.gnu.org
Hi,
raid5atemyhomework <raid5atemyhomework@protonmail.com> skribis:
>>From 4beb73c62995cf236b402dad8e1c36016027c781 Mon Sep 17 00:00:00 2001
> From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
> Date: Tue, 5 Jan 2021 22:27:56 +0800
> Subject: [PATCH 1/4] gnu: Allow services to install kernel-loadable modules.
>
> * gnu/system.scm (operating-system-directory-base-entries): Remove code
> to handle generation of "kernel" and "hurd".
> (operating-system-default-essential-services): Instantiate
> kernel-loadable-module-service.
> (hurd-default-essential-services): Instantiate
> kernel-loadable-module-service.
> (package-for-kernel): Move ...
> * gnu/services.scm: ... to here.
> (kernel-loadable-module-service-type): New variable.
> (kernel-loadable-module-service): New procedure.
> * gnu/tests/linux-modules.scm (run-loadable-kernel-modules-test): Move
> code to ...
> (run-loadable-kernel-modules-test-base): ... new procedure here.
> (run-loadable-kernel-modules-service-test): New procedure.
> (%test-loadable-kernel-modules-service-0): New variable.
> (%test-loadable-kernel-modules-service-1): New variable.
> (%test-loadable-kernel-modules-service-2): New variable.
> * doc/guix.texi: Document kernel-loadable-module-service-type.
[…]
> +@defvr {Scheme Variable} kernel-loadable-module-service-type
> +Type of the service that collects lists of packages containing
> +kernel-loadable modules, and adds them to the set of kernel-loadable
> +modules.
> +@end defvr
Would be nice to expound a bit here, in particular by adding an example
(along the lines of those used in system tests maybe?). Otherwise it
can be hard to fathom how this is meant to be used.
> +;; Configuration for the kernel builder.
> +(define-record-type* <kernel-builder-configuration> kernel-builder-configuration
> + make-kernel-builder-configuration
> + kernel-builder-configuration?
> + this-kernel-builder-configuration
> +
> + (kernel kernel-builder-configuration-kernel (default #f))
> + (hurd kernel-builder-configuration-hurd (default #f))
> + (modules kernel-builder-configuration-modules (default '())))
How about <linux-build-configuration> instead?
In general, throughout the project, we do not use “kernel” and “Linux”
interchangeably. Since this is a Linux-only feature, let’s call it that
way and remove the ‘hurd’ field (the Hurd has no notion of in-kernel
modules since pretty much everything happens in user-space.)
> +(define kernel-loadable-module-service-type
> + (service-type (name 'kernel-loadable-modules)
Same here: ‘linux-loadable-module-service-type’.
But… it’s not clear at first sight how this differs from the existing
‘kernel-module-loader’. Perhaps ‘linux-build-service-type’ would be
more accurate? Or am I missing something?
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
2021-02-10 15:44 ` Ludovic Courtès
@ 2021-02-10 16:49 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-02-10 16:49 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: Danny Milosavljevic, 45692@debbugs.gnu.org
> > +;; Configuration for the kernel builder.
> > +(define-record-type* <kernel-builder-configuration> kernel-builder-configuration
> >
> > - make-kernel-builder-configuration
> > - kernel-builder-configuration?
> > - this-kernel-builder-configuration
> > -
> > - (kernel kernel-builder-configuration-kernel (default #f))
> > - (hurd kernel-builder-configuration-hurd (default #f))
> > - (modules kernel-builder-configuration-modules (default '())))
>
> How about <linux-build-configuration> instead?
>
> In general, throughout the project, we do not use “kernel” and “Linux”
> interchangeably. Since this is a Linux-only feature, let’s call it that
> way and remove the ‘hurd’ field (the Hurd has no notion of in-kernel
> modules since pretty much everything happens in user-space.)
The `operating-system` record uses `kernel-loadable-modules` as the record field name. I suggest changing that first, if you truly want to differentiate "kernel" from "linux" "throughout the project". Or deprecate it entirely and instead use the new `linux-loadable-modules-service-type`, in principle the only field needed in `operating-system` should be `services`.
>
> > +(define kernel-loadable-module-service-type
> >
> > - (service-type (name 'kernel-loadable-modules)
>
> Same here: ‘linux-loadable-module-service-type’.
>
> But… it’s not clear at first sight how this differs from the existing
> ‘kernel-module-loader’. Perhaps ‘linux-build-service-type’ would be
> more accurate? Or am I missing something?
`kernel-module-loader` explicitly loads a module at startup, it does not make a non-Linux-libre-built-in module actually *loadable*. So there is a need for something to augment the `linux-loadable-modules` record field of `operating-system`.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix
2021-02-10 14:27 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
2021-02-10 14:32 ` raid5atemyhomework via Guix-patches via
@ 2021-02-13 1:49 ` raid5atemyhomework via Guix-patches via
1 sibling, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-02-13 1:49 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 45692@debbugs.gnu.org
Hello Ludo',
> > +As a large and complex kernel module, ZFS on Linux has to be compiled
>
> ^
>
>
> I think it’s a proper noun and is spelled “ZFS-on-Linux”.
FWIW ZFSonLinux is now folded into OpenZFS (which is now a package that includes both Linux and FreeBSD support, with plans to merge in the Illumos codebase as well) since 2.0.0. The documentation was written when Guix was still at 0.8.5 (when it was definitely ZoL), so I should probably change this to OpenZFS, and clarify whether I am referring to the ZFS file system or the OpenZFS package.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 0/3] New patch series for Even Better ZFS Support on Guix
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (3 preceding siblings ...)
2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
@ 2021-03-22 14:33 ` raid5atemyhomework via Guix-patches via
2021-03-28 12:55 ` Léo Le Bouter via Guix-patches via
2021-03-22 14:33 ` [bug#45692] [PATCH v3 1/3] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
` (7 subsequent siblings)
12 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-03-22 14:33 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
This is a new patch series, please completely forget the previous patch series.
Notable differences are:
* ZFS automounted datasets are mounted before `user-processes` instead of `file-systems`, since there is strong objection to modifying `file-systems` so it can be extended to allow ZFS automounted datasets to be mounted before `file-systems`. This prevents easy `/home` on ZFS (though see the added documentation in 3/3 for a workaround).
* The old `kernel-loadable-module-service-type` is renamed to `linux-loadable-module-service-type`, despite the corresponding `operating-system` field being named `kernel-loadable-modules`, and the loader service type being named `kernel-module-loader-service-type` :rolleyes:.
* There is a new `linux-builder-service-type`, which can be extended with a function to modify its `linux-builder-configuration`. The `linux-loadable-module-service-type` extends this base service. I think this design patttern should be more extensively used in Guix; currently services are only extensible via one field of their configuration, this pattern allows creating multiple service types for each extensible field of the configuration. In this particular case, the `kernel` of the `linux-builder-service-type` can be extended (by adding/removing `configure` or `make` flags, or added `inputs`) independently of the list of loadable modules using this design pattern.
* Auto-scrub and auto-snapshot is now supported, with additional fields in `zfs-configuration` to support it.
* The auto-mounting Shepherd service has been renamed `zfs-auto-mount` (previously `zfs-automount`).
* Documentation includes a list of things that are currently not (fully) supported.
Personally I'm not particularly motivated to push this anymore. I have a frankenstein's `configuration.scm` that gets me the bits of ZFS functionality that I absolutely need, with the rest something I'm just living without, and while it would be *nice* to have *some* out-of-the-box support for ZFS (or at least make it easy to get a box to get support for ZFS out of), I personally don't need it at this point.
Patch 1/3 corresponds to the original Patch 1/4. Patch 2/3 has no corresponding patch in the original patch set, but is the same patch as https://issues.guix.gnu.org/47134 which nobody is looking at, as usual. Patch 3/3 corresponds to the original Patch 4/4.
The original Patch 2/4 has been removed since it's what makes `/home` on ZFS work by making `file-systems` extensible. The original Patch 3/4 was already merged.
This patch series is minimally tested compared to the previous patch series; I have homework to store on my ZFS pool and should really stop fooling around with the operating system. I have this patch series running in a VM with a ZFS pool. Auto-scrub is not tested, though auto-snapshot looks working. Auto-mounting works as with previous patch series. Encrypted and compressed filesystems are untested but I don't expect it to be significantly regressed compared to the previous patch series.
More work can be done for ZFS support. But I think the current set gives a workable ZFS support; further work can be done piecemeal afterwards. The documentation includes notes for anyone who feels motivated enough to keep improving the ZFS support.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 1/3] gnu: Allow services to install kernel-loadable modules.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (4 preceding siblings ...)
2021-03-22 14:33 ` [bug#45692] [PATCH v3 0/3] New patch series for " raid5atemyhomework via Guix-patches via
@ 2021-03-22 14:33 ` raid5atemyhomework via Guix-patches via
2021-05-11 14:17 ` Danny Milosavljevic
2021-03-22 14:34 ` [bug#45692] [PATCH v3 2/3] gnu: Add zfs-auto-snapshot raid5atemyhomework via Guix-patches via
` (6 subsequent siblings)
12 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-03-22 14:33 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
From d54d718dd83195041d9f536e8c675eb2bffdcb8d Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Mon, 22 Mar 2021 11:23:32 +0800
Subject: [PATCH 1/3] gnu: Allow services to install kernel-loadable modules.
* gnu/system.scm (operating-system-directory-base-entries): Remove code
to handle generation of "kernel" for linux-libre kernels.
(operating-system-default-essential-services): Instantiate
linux-builder-service-type.
(package-for-kernel): Move ...
* gnu/services.scm: ... to here.
(linux-builder-service-type): New variable.
(linux-builder-configuration): New type.
(linux-loadable-module-service-type): New variable.
* gnu/tests/linux-modules.scm (run-loadable-kernel-modules-test): Move
code to ...
(run-loadable-kernel-modules-test-base): ... new procedure here.
(run-loadable-kernel-modules-service-test): New procedure.
(%test-loadable-kernel-modules-service-0): New variable.
(%test-loadable-kernel-modules-service-1): New variable.
(%test-loadable-kernel-modules-service-2): New variable.
* doc/guix.texi: Document linux-loadable-module-service-type.
---
doc/guix.texi | 22 +++++++++
gnu/services.scm | 90 +++++++++++++++++++++++++++++++++++++
gnu/system.scm | 34 ++++----------
gnu/tests/linux-modules.scm | 80 ++++++++++++++++++++++++++++-----
4 files changed, 191 insertions(+), 35 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 386169b2a5..86b22f3673 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -33967,6 +33967,28 @@ configuration when you use @command{guix system reconfigure},
@command{guix system init}, or @command{guix deploy}.
@end defvr
+@defvr {Scheme Variable} linux-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+
+This service type is intended to be extended by other service types,
+such as below:
+
+@lisp
+(define module-installing-service-type
+ (service-type
+ (name 'module-installing-service)
+ (extensions (list (service-extension linux-loadable-module-service-type
+ (const (list module-to-install-1
+ module-to-install-2)))))
+ (default-value #f)))
+@end lisp
+
+This does not actually load modules at bootup, only adds it to the
+kernel profile so that it @emph{can} be loaded by other means.
+@end defvr
+
@node Shepherd Services
@subsection Shepherd Services
diff --git a/gnu/services.scm b/gnu/services.scm
index ddd1bac30c..a20edeb8ec 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2016 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -33,6 +34,8 @@
#:use-module (guix diagnostics)
#:autoload (guix openpgp) (openpgp-format-fingerprint)
#:use-module (guix modules)
+ #:use-module (guix packages)
+ #:use-module (guix utils)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages hurd)
@@ -106,6 +109,12 @@
profile-service-type
firmware-service-type
gc-root-service-type
+ linux-builder-service-type
+ linux-builder-configuration
+ linux-builder-configuration?
+ linux-builder-configuration-kernel
+ linux-builder-configuration-modules
+ linux-loadable-module-service-type
%boot-service
%activation-service
@@ -872,6 +881,87 @@ as Wifi cards.")))
will not be reclaimed by the garbage collector.")
(default-value '())))
+;; Configuration for the Linux kernel builder.
+(define-record-type* <linux-builder-configuration>
+ linux-builder-configuration
+ make-linux-builder-configuration
+ linux-builder-configuration?
+ this-linux-builder-configuration
+
+ (kernel linux-builder-configuration-kernel) ; package
+ (modules linux-builder-configuration-modules (default '()))) ; list of packages
+
+(define (package-for-kernel target-kernel module-package)
+ "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
+possible (that is if there's a LINUX keyword argument in the build system)."
+ (package
+ (inherit module-package)
+ (arguments
+ (substitute-keyword-arguments (package-arguments module-package)
+ ((#:linux kernel #f)
+ target-kernel)))))
+
+(define (linux-builder-configuration->system-entry config)
+ "Return the kernel entry of the 'system' directory."
+ (let* ((kernel (linux-builder-configuration-kernel config))
+ (modules (linux-builder-configuration-modules config))
+ (kernel (profile
+ (content (packages->manifest
+ (cons kernel
+ (map (lambda (module)
+ (cond
+ ((package? module)
+ (package-for-kernel kernel module))
+ ;; support (,package "kernel-module-output")
+ ((and (list? module) (package? (car module)))
+ (cons (package-for-kernel kernel
+ (car module))
+ (cdr module)))
+ (else
+ module)))
+ modules))))
+ (hooks (list linux-module-database)))))
+ (with-monad %store-monad
+ (return `(("kernel" ,kernel))))))
+
+(define linux-builder-service-type
+ (service-type (name 'linux-builder)
+ (extensions
+ (list (service-extension system-service-type
+ linux-builder-configuration->system-entry)))
+ (default-value '())
+ (compose identity)
+ (extend (lambda (config modifiers)
+ (if (null? modifiers)
+ config
+ ((apply compose modifiers) config))))
+ (description "Builds the linux-libre kernel profile, containing
+the kernel itself and any linux-loadable kernel modules. This can be extended
+with a function that accepts the current configuration and returns a new
+configuration.")))
+
+(define (linux-loadable-module-builder-modifier modules)
+ "Extends linux-builder-service-type by appending the given MODULES to the
+configuration of linux-builder-service-type."
+ (lambda (config)
+ (linux-builder-configuration
+ (inherit config)
+ (modules (append (linux-builder-configuration-modules config)
+ modules)))))
+
+(define linux-loadable-module-service-type
+ (service-type (name 'linux-loadable-modules)
+ (extensions
+ (list (service-extension linux-builder-service-type
+ linux-loadable-module-builder-modifier)))
+ (default-value '())
+ (compose concatenate)
+ (extend append)
+ (description "Adds packages and package outputs as modules
+included in the booted linux-libre profile. Other services can extend this
+service type to add particular modules to the set of linux-loadable modules.")))
+
+
\f
;;;
;;; Service folding.
diff --git a/gnu/system.scm b/gnu/system.scm
index 5bf2a85272..7cc4f134b7 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -13,6 +13,7 @@
;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <jannek@gnu.org>
;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -601,16 +602,6 @@ OS."
(file-append (operating-system-kernel os)
"/" (system-linux-image-file-name))))
-(define (package-for-kernel target-kernel module-package)
- "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
-possible (that is if there's a LINUX keyword argument in the build system)."
- (package
- (inherit module-package)
- (arguments
- (substitute-keyword-arguments (package-arguments module-package)
- ((#:linux kernel #f)
- target-kernel)))))
-
(define %default-modprobe-blacklist
;; List of kernel modules to blacklist by default.
'("usbmouse" ;races with bcm5974, see <https://bugs.gnu.org/35574>
@@ -628,23 +619,12 @@ value of the SYSTEM-SERVICE-TYPE service."
(let* ((locale (operating-system-locale-directory os))
(kernel (operating-system-kernel os))
(hurd (operating-system-hurd os))
- (modules (operating-system-kernel-loadable-modules os))
- (kernel (if hurd
- kernel
- (profile
- (content (packages->manifest
- (cons kernel
- (map (lambda (module)
- (if (package? module)
- (package-for-kernel kernel
- module)
- module))
- modules))))
- (hooks (list linux-module-database)))))
(initrd (and (not hurd) (operating-system-initrd-file os)))
(params (operating-system-boot-parameters-file os)))
- `(("kernel" ,kernel)
- ,@(if hurd `(("hurd" ,hurd)) '())
+ `(,@(if hurd
+ `(("hurd" ,hurd)
+ ("kernel" ,kernel))
+ '())
("parameters" ,params)
,@(if initrd `(("initrd" ,initrd)) '())
("locale" ,locale)))) ;used by libc
@@ -664,6 +644,10 @@ bookkeeping."
(host-name (host-name-service (operating-system-host-name os)))
(entries (operating-system-directory-base-entries os)))
(cons* (service system-service-type entries)
+ (service linux-builder-service-type
+ (linux-builder-configuration
+ (kernel (operating-system-kernel os))
+ (modules (operating-system-kernel-loadable-modules os))))
%boot-service
;; %SHEPHERD-ROOT-SERVICE must come last so that the gexp that
diff --git a/gnu/tests/linux-modules.scm b/gnu/tests/linux-modules.scm
index 953b132ef7..30d8eae03b 100644
--- a/gnu/tests/linux-modules.scm
+++ b/gnu/tests/linux-modules.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2019 Jakob L. Kreuze <zerodaysfordays@sdf.org>
;;; Copyright © 2020 Danny Milosavljevic <dannym@scratchpost.org>
;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -34,7 +35,10 @@
#:use-module (guix utils)
#:export (%test-loadable-kernel-modules-0
%test-loadable-kernel-modules-1
- %test-loadable-kernel-modules-2))
+ %test-loadable-kernel-modules-2
+ %test-loadable-kernel-modules-service-0
+ %test-loadable-kernel-modules-service-1
+ %test-loadable-kernel-modules-service-2))
;;; Commentary:
;;;
@@ -66,17 +70,11 @@ that MODULES are actually loaded."
(member module modules string=?))
'#$modules))))))
-(define* (run-loadable-kernel-modules-test module-packages module-names)
- "Run a test of an OS having MODULE-PACKAGES, and verify that MODULE-NAMES
-are loaded in memory."
+(define* (run-loadable-kernel-modules-test-base base-os module-names)
+ "Run a test of BASE-OS, verifying that MODULE-NAMES are loaded in memory."
(define os
(marionette-operating-system
- (operating-system
- (inherit (simple-operating-system))
- (services (cons (service kernel-module-loader-service-type module-names)
- (operating-system-user-services
- (simple-operating-system))))
- (kernel-loadable-modules module-packages))
+ base-os
#:imported-modules '((guix combinators))))
(define vm (virtual-machine os))
(define (test script)
@@ -98,6 +96,36 @@ are loaded in memory."
(gexp->derivation "loadable-kernel-modules"
(test (modules-loaded?-program os module-names))))
+(define* (run-loadable-kernel-modules-test module-packages module-names)
+ "Run a test of an OS having MODULE-PACKAGES, and verify that MODULE-NAMES
+are loaded in memory."
+ (run-loadable-kernel-modules-test-base
+ (operating-system
+ (inherit (simple-operating-system))
+ (services (cons (service kernel-module-loader-service-type module-names)
+ (operating-system-user-services
+ (simple-operating-system))))
+ (kernel-loadable-modules module-packages))
+ module-names))
+
+(define* (run-loadable-kernel-modules-service-test module-packages module-names)
+ "Run a test of an OS having MODULE-PACKAGES, which are loaded by creating a
+service that extends LINUXL-LOADABLE-MODULE-SERVICE-TYPE. Then verify that
+MODULE-NAMES are loaded in memory."
+ (define module-installing-service-type
+ (service-type
+ (name 'module-installing-service)
+ (extensions (list (service-extension linux-loadable-module-service-type
+ (const module-packages))))
+ (default-value #f)))
+ (run-loadable-kernel-modules-test-base
+ (operating-system
+ (inherit (simple-operating-system))
+ (services (cons* (service module-installing-service-type)
+ (operating-system-user-services
+ (simple-operating-system)))))
+ module-names))
+
(define %test-loadable-kernel-modules-0
(system-test
(name "loadable-kernel-modules-0")
@@ -129,3 +157,35 @@ with two extra modules.")
(package-arguments
ddcci-driver-linux))))))
'("acpi_call" "ddcci")))))
+
+(define %test-loadable-kernel-modules-service-0
+ (system-test
+ (name "loadable-kernel-modules-service-0")
+ (description "Tests loadable kernel modules extensible service with no
+extra modules.")
+ (value (run-loadable-kernel-modules-service-test '() '()))))
+
+(define %test-loadable-kernel-modules-service-1
+ (system-test
+ (name "loadable-kernel-modules-service-1")
+ (description "Tests loadable kernel modules extensible service with one
+extra module.")
+ (value (run-loadable-kernel-modules-service-test
+ (list ddcci-driver-linux)
+ '("ddcci")))))
+
+(define %test-loadable-kernel-modules-service-2
+ (system-test
+ (name "loadable-kernel-modules-service-2")
+ (description "Tests loadable kernel modules extensible service with two
+extra modules.")
+ (value (run-loadable-kernel-modules-service-test
+ (list acpi-call-linux-module
+ (package
+ (inherit ddcci-driver-linux)
+ (arguments
+ `(#:linux #f
+ ,@(strip-keyword-arguments '(#:linux)
+ (package-arguments
+ ddcci-driver-linux))))))
+ '("acpi_call" "ddcci")))))
--
2.31.0
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 2/3] gnu: Add zfs-auto-snapshot.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (5 preceding siblings ...)
2021-03-22 14:33 ` [bug#45692] [PATCH v3 1/3] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
@ 2021-03-22 14:34 ` raid5atemyhomework via Guix-patches via
2021-05-11 14:05 ` Danny Milosavljevic
2021-03-22 14:35 ` [bug#45692] [PATCH v3 3/3] gnu: Add ZFS service type raid5atemyhomework via Guix-patches via
` (5 subsequent siblings)
12 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-03-22 14:34 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
From 481dc87f77743b7a282777656e78615ea57d19cc Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Sun, 14 Mar 2021 16:40:47 +0800
Subject: [PATCH 2/3] gnu: Add zfs-auto-snapshot.
* gnu/packages/file-systems.scm (zfs-auto-snapshot): New variable.
---
gnu/packages/file-systems.scm | 75 +++++++++++++++++++++++++++++++++++
1 file changed, 75 insertions(+)
diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 198653c639..6877f89f6e 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -976,6 +976,81 @@ originally developed for Solaris and is now maintained by the OpenZFS
community.")
(license license:cddl1.0)))
+(define-public zfs-auto-snapshot
+ (package
+ (name "zfs-auto-snapshot")
+ (version "1.2.4")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/zfsonlinux/zfs-auto-snapshot/archive/upstream/"
+ version ".tar.gz"))
+ (sha256
+ (base32 "16ry1w43i44xc67gr73x6fa48ninfhqxr498ad4m3kya93vp2zrh"))))
+ (build-system gnu-build-system)
+ (inputs
+ ;; Note: if you are inheriting from the above zfs package in order
+ ;; to provide a specific stable kernel version, you should also
+ ;; inherit this package and replace the sole input below.
+ `(("zfs" ,zfs)))
+ (arguments
+ `(#:tests? #f ; No tests
+ #:phases
+ (modify-phases %standard-phases
+ (delete 'configure)
+ (delete 'build)
+ ;; Guix System may not have a traditional cron system, but
+ ;; the cron scripts installed by this package are convenient
+ ;; to use as targets for an mcron job specification, so make
+ ;; sure they can be run in-store.
+ (add-before 'install 'fix-scripts
+ (lambda* (#:key outputs inputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (zfs-auto-snapshot (string-append
+ out
+ "/sbin/zfs-auto-snapshot"))
+ (zfs-package (assoc-ref inputs "zfs"))
+ (zpool (string-append
+ zfs-package
+ "/sbin/zpool"))
+ (zfs (string-append
+ zfs-package
+ "/sbin/zfs")))
+ (substitute* '("etc/zfs-auto-snapshot.cron.daily"
+ "etc/zfs-auto-snapshot.cron.frequent"
+ "etc/zfs-auto-snapshot.cron.hourly"
+ "etc/zfs-auto-snapshot.cron.monthly"
+ "etc/zfs-auto-snapshot.cron.weekly")
+ (("zfs-auto-snapshot")
+ zfs-auto-snapshot))
+ (substitute* "src/zfs-auto-snapshot.sh"
+ (("LC_ALL=C zfs list")
+ (string-append "LC_ALL=C " zfs " list"))
+ (("LC_ALL=C zpool status")
+ (string-append "LC_ALL=C " zpool " status"))
+ (("zfs snapshot")
+ (string-append zfs " snapshot"))
+ (("zfs destroy")
+ (string-append zfs " destroy"))))))
+ ;; Provide DESTDIR and PREFIX on make command.
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let ((out (assoc-ref outputs "out")))
+ (invoke "make" "install"
+ "PREFIX="
+ (string-append "DESTDIR=" out)))
+ #t)))))
+ (home-page "https://github.com/zfsonlinux/zfs-auto-snapshot")
+ (synopsis "Automatically create, rotate, and destroy periodic ZFS snapshots")
+ (description
+ "An alternative implementation of the zfs-auto-snapshot service for Linux
+that is compatible with zfs-linux (now OpenZFS) and zfs-fuse.
+
+On Guix System, you will need to invoke the included shell scripts as @code{job}
+definitions in your @code{operating-system} declaration.")
+ (license license:gpl2+)))
+
(define-public mergerfs
(package
(name "mergerfs")
--
2.31.0
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 3/3] gnu: Add ZFS service type.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (6 preceding siblings ...)
2021-03-22 14:34 ` [bug#45692] [PATCH v3 2/3] gnu: Add zfs-auto-snapshot raid5atemyhomework via Guix-patches via
@ 2021-03-22 14:35 ` raid5atemyhomework via Guix-patches via
2021-07-25 14:03 ` raid5atemyhomework via Guix-patches via
2021-07-25 14:31 ` [bug#45692] [PATCH v4 " raid5atemyhomework via Guix-patches via
` (4 subsequent siblings)
12 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-03-22 14:35 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
From 7c039aee628bdb0188c0da27defb591f7791053e Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Mon, 22 Mar 2021 16:26:28 +0800
Subject: [PATCH 3/3] gnu: Add ZFS service type.
* gnu/services/file-systems.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
* gnu/services/base.scm: Export dependency->shepherd-service-name.
* doc/guix.texi (ZFS File System): New subsection.
---
doc/guix.texi | 351 ++++++++++++++++++++++++++++++++++
gnu/local.mk | 2 +
gnu/services/base.scm | 4 +-
gnu/services/file-systems.scm | 295 ++++++++++++++++++++++++++++
4 files changed, 651 insertions(+), 1 deletion(-)
create mode 100644 gnu/services/file-systems.scm
diff --git a/doc/guix.texi b/doc/guix.texi
index 86b22f3673..a2a092b31b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -87,6 +87,7 @@ Copyright @copyright{} 2020 Daniel Brooks@*
Copyright @copyright{} 2020 John Soo@*
Copyright @copyright{} 2020 Jonathan Brielmaier@*
Copyright @copyright{} 2020 Edgar Vincent@*
+Copyright @copyright{} 2021 raid5atemyhomework@*
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -14095,6 +14096,356 @@ a file system declaration such as:
compress-force=zstd,space_cache=v2"))
@end lisp
+
+@node ZFS File System
+@subsection ZFS File System
+
+Support for ZFS file systems is provided on Guix by the OpenZFS project.
+OpenZFS currently only supports Linux-Libre and is not available on the
+Hurd.
+
+OpenZFS is free software; unfortunately its license is incompatible with
+the GNU General Public License (GPL), the license of the Linux kernel,
+which means they cannot be distributed together. However, as a user,
+you can choose to build ZFS and use it together with Linux; you can
+even rely on Guix to automate this task. See
+@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
+the Free Software Foundation} for more information.
+
+As a large and complex kernel module, OpenZFS has to be compiled for a
+specific version of Linux-Libre. At times, the latest OpenZFS package
+available in Guix is not compatible with the latest Linux-Libre version.
+Thus, directly installing the @code{zfs} package can fail.
+
+Instead, you are recommended to select a specific older long-term-support
+Linux-Libre kernel. Do not use @code{linux-libre-lts}, as even the
+latest long-term-support kernel may be too new for @code{zfs}. Instead,
+explicitly select a specific older version, such as @code{linux-libre-5.10},
+and upgrade it manually later as new long-term-support kernels become
+available that you have confirmed is compatible with the latest available
+OpenZFS version on Guix.
+
+For example, you can modify your system configuration file to a specific
+Linux-Libre version and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+ #;@dots{}
+ linux)
+(use-service-modules
+ #;@dots{}
+ file-systems)
+
+(define my-kernel linux-libre-5.10)
+
+(operating-system
+ (kernel my-kernel)
+ #;@dots{}
+ (services
+ (cons* (service zfs-service-type
+ (zfs-configuration
+ (kernel my-kernel)))
+ #;@dots{}
+ %desktop-services))
+ #;@dots{})
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type for a service that adds ZFS support to your operating
+system. The service is configured using a @code{zfs-configuration}
+record.
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+ (zfs-configuration
+ (kernel linux-libre-5.4)))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS support in Guix
+System. Its fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux-Libre kernel to compile OpenZFS for. This field
+is always required. It @emph{must} be the same kernel you use in your
+@code{operating-system} form.
+
+@item @code{base-zfs} (default: @code{zfs})
+The OpenZFS package that will be compiled for the given Linux-Libre kernel.
+
+@item @code{base-zfs-auto-snapshot} (default: @code{zfs-auto-snapshot})
+The @code{zfs-auto-snapshot} package to use. It will be modified to
+specifically use the OpenZFS compiled for your kernel.
+
+@item @code{dependencies} (default: @code{'()})
+A list of @code{<file-system>} or @code{<mapped-device>} records that must
+be mounted or opened before OpenZFS scans for pools to import. For example,
+if you have set up LUKS containers as leaf VDEVs in a pool, you have to
+include their corresponding @code{<mapped-ddevice>} records so that OpenZFS
+can import the pool correctly at bootup.
+
+@item @code{auto-mount?} (default: @code{#t})
+Whether to mount datasets with the ZFS @code{mountpoint} property automatically
+at startup. This is the behavior that ZFS users usually expect. You might
+set this to @code{#f} for an operating system intended as a ``rescue'' system
+that is intended to help debug problems with the disks rather than actually
+work in production.
+
+@item @code{auto-scrub} (default: @code{'weekly})
+Specifies how often to scrub all pools. Can be the symbols @code{'weekly} or
+@code{'monthly}, or a schedule specification understood by
+@xref{mcron, mcron job specifications,, mcron, GNU@tie{}mcron}, such as
+@code{"0 3 * * 6"} for ``every 3AM on Saturday''.
+It can also be @code{#f} to disable auto-scrubbing (@strong{not recommended}).
+
+The general guideline is to scrub weekly when using consumer-quality drives, and
+to scrub monthly when using enterprise-quality drives.
+
+@code{'weekly} scrubs are done on Sunday midnight, while @code{monthly} scrubs
+are done on midnight on the first day of each month.
+
+@item @code{auto-snapshot?} (default: @code{#t})
+Specifies whether to auto-snapshot by default. If @code{#t}, then snapshots
+are automatically created except for ZFS datasets with the
+@code{com.sun:auto-snapshot} ZFS vendor property set to @code{false}.
+
+If @code{#f}, snapshots will not be automatically created, unless the ZFS
+dataset has the @code{com.sun:auto-snapshot} ZFS vendor property set to
+@code{true}.
+
+@item @code{auto-snapshot-keep} (default: @code{'()})
+Specifies an association list of symbol-number pairs, indicating the number
+of automatically-created snapshots to retain for each frequency type.
+
+If not specified via this field, by default there are 4 @code{frequent}, 24
+@code{hourly}, 31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly} snapshots.
+
+For example:
+
+@lisp
+(zfs-configuration
+ (kernel my-kernel)
+ (auto-snapshot-keep
+ '((frequent . 8)
+ (hourly . 12))))
+@end lisp
+
+The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
+@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
+defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
+
+@end table
+@end deftp
+
+@subsubsection ZFS Auto-Snapshot
+
+The ZFS service on Guix System supports auto-snapshots as implemented in the
+Solaris operating system.
+
+@code{frequent} (every 15 minutes), @code{hourly}, @code{daily}, @code{weekly},
+and @code{monthly} snapshots are created automatically for ZFS datasets that
+have auto-snapshot enabled. They will be named, for example,
+@code{zfs-auto-snap_frequent-2021-03-22-1415}. You can continue to use
+manually-created snapshots as long as they do not conflict with the naming
+convention used by auto-snapshot. You can also safely manually destroy
+automatically-created snapshots, for example to free up space.
+
+The @code{com.sun:auto-snapshot} ZFS property controls auto-snapshot on a
+per-dataset level. Sub-datasets will inherit this property from their parent
+dataset, but can have their own property.
+
+You @emph{must} set this property to @code{true} or @code{false} exactly,
+otherwise it will be treated as if the property is unset.
+
+For example:
+
+@example
+# zfs list -o name
+NAME
+tank
+tank/important-data
+tank/tmp
+# zfs set com.sun:auto-snapshot=true tank
+# zfs set com.sun:auto-snapshot=false tank/tmp
+@end example
+
+The above will set @code{tank} and @code{tank/important-data} to be
+auto-snapshot, while @code{tank/tmp} will not be auto-snapshot.
+
+If the @code{com.sun:auto-snapshot} property is not set for a dataset
+(the default when pools and datasets are created), then whether
+auto-snapshot is done or not will depend on the @code{auto-snapshot?}
+field of the @code{zfs-configuration} record.
+
+There are also @code{com.sun:auto-snapshot:frequent},
+@code{com.sun:auto-snapshot:hourly}, @code{com.sun:auto-snapshot:daily},
+@code{com.sun:auto-snapshot:weekly}, and @code{com.sun:auto-snapshot:monthly}
+properties that give finer-grained control of whether to auto-snapshot a
+dataset at a particular schedule.
+
+The number of snapshots kept for all datasets can be overridden via the
+@code{auto-snapshot-keep} field of the @code{zfs-configuration} record.
+There is currently no support to have different numbers of snapshots to
+keep for different datasets.
+
+@subsubsection ZVOLs
+
+ZFS supports ZVOLs, block devices that ZFS exposes to the operating
+system in the @code{/dev/zvol/} directory. The ZVOL will have the same
+resilience and self-healing properties as other datasets on your ZFS pool.
+ZVOLs can also be snapshotted (and will be included in auto-snapshotting
+if enabled), which snapshots the state of the block device, effectively
+snapshotting the hosted file system.
+
+You can put any file system inside the ZVOL. However, in order to mount this
+file system at system start, you need to add @code{%zfs-zvol-dependency} as a
+dependency of each file system inside a ZVOL.
+
+@defvr {Scheme Variable} %zfs-zvol-dependency
+An artificial @code{<mapped-device>} which tells the file system mounting
+service to wait for ZFS to provide ZVOLs before mounting the
+@code{<file-system>} dependent on it.
+@end defvr
+
+For example, suppose you create a ZVOL and put an ext4 filesystem
+inside it:
+
+@example
+# zfs create -V 100G tank/ext4-on-zfs
+# mkfs.ext4 /dev/zvol/tank/ext4-on-zfs
+# mkdir /ext4-on-zfs
+# mount /dev/zvol/tank/ext4-on-zfs /ext4-on-zfs
+@end example
+
+You can then set this up to be mounted at boot by adding this to the
+@code{file-systems} field of your @code{operating-system} record:
+
+@lisp
+(file-system
+ (device "/dev/zvol/tank/ext4-on-zfs")
+ (mount-point "/ext4-on-zfs")
+ (type "ext4")
+ (dependencies (list %zfs-zvol-dependency)))
+@end lisp
+
+You @emph{must not} add @code{%zfs-zvol-dependency} to your
+@code{operating-system}'s @code{mapped-devices} field, and you @emph{must
+not} add it (or any @code{<file-system>}s dependent on it) to the
+@code{dependencies} field of @code{zfs-configuration}. Finally, you
+@emph{must not} use @code{%zfs-zvol-dependency} unless you actually
+instantiate @code{zfs-service-type} on your system.
+
+@subsubsection Unsupported Features
+
+Some common features and uses of ZFS are currently not supported, or not
+fully supported, on Guix.
+
+@enumerate
+@item
+Shepherd-managed daemons that are configured to read from or write to ZFS
+mountpoints need to include @code{user-processes} in their @code{requirement}
+field. This is the earliest that ZFS file systems are assured of being
+mounted.
+
+Generally, most daemons will, directly or indirectly, require
+@code{networking}, or @code{user-processes}, or both. Most implementations
+of @code{networking} will also require @code{user-processes} so daemons that
+require only @code{networking} will also generally start up after
+@code{user-processes}. A notable exception, however, is
+@code{static-networking-service-type}. You will need to explicitly add
+@code{user-processes} as a @code{requirement} of your @code{static-networking}
+record.
+
+@item
+@code{mountpoint=legacy} ZFS file systems. The handlers for the Guix mounting
+system have not yet been modified to support ZFS, and will expect @code{/dev}
+paths in the @code{<file-system>}'s @code{device} field, but ZFS file systems
+are referred to via non-path @code{pool/file/system} names. Such file systems
+also need to be mounted @emph{after} OpenZFS has scanned for pools.
+
+You can still manually mount these file systems after system boot; what is
+only unsupported is mounting them automatically at system boot by specifying
+them in @code{<file-system>} records of your @code{operating-system}.
+
+@item
+@code{/home} on ZFS. Guix will create home directories for users, but this
+process currently cannot be scheduled after ZFS file systems are mounted.
+Thus, the ZFS file system might be mounted @emph{after} Guix has created
+home directories at boot, at which point OpenZFS will refuse to mount since
+the mountpoint is not empty. However, you @emph{can} create an ext4, xfs,
+btrfs, or other supported file system inside a ZVOL, have that depend on
+@code{%zfs-zvol-dependency}, and set it to mount on the @code{/home}
+directory; they will be scheduled to mount before the @code{user-homes}
+process.
+
+Similarly, other locations like @code{/var}, @code{/gnu/store} and so
+on cannot be reliably put in a ZFS file system, though they may be
+possible to create as other file systems inside ZVOL containers.
+
+@item
+@code{/} and @code{/boot} on ZFS. These require Guix to expose more of
+the @code{initrd} very early boot process to services. It also requires
+Guix to have the ability to explicitly load modules while still in
+@code{initrd} (currently kernel modules loaded by
+@code{kernel-module-loader-service-type} are loaded after @code{/} is
+mounted). Further, since one of ZFS's main advantages is that it can
+continue working despite the loss of one or more devices, it makes sense
+to also support installing the bootloader on all devices of the pool that
+contains the @code{/} and @code{/boot}; after all, if ZFS can survive the
+loss of one device, the bootloader should also be able to survive the loss
+of one device.
+
+@item
+ZVOL swap devices. Mapped swap devices need to be listed in
+@code{mapped-devices} to ensure they are opened before the system attempts
+to use them, but you cannot currently add @code{%zfs-zvol-dependency} to
+@code{mapped-devices}.
+
+This will also require significant amounts of testing, as various kernel
+build options and patches may affect how swapping works, which are possibly
+different on Guix System compared to other distributions that this feature is
+known to work on.
+
+@item
+ZFS Event Daemon. Support for this has not been written yet, patches are
+welcome. The main issue is how to design this in a Guix style while
+supporting legacy shell-script styles as well. In particular, OpenZFS itself
+comes with a number of shell scripts intended for ZFS Event Daemon, and we
+need to figure out how the user can choose to use or not use the provided
+scripts (and configure any settings they have) or override with their own
+custom code (which could be shell scripts they have written and trusted from
+previous ZFS installations).
+
+As-is, you can create your own service that activates the ZFS Event Daemon
+by creating the @file{/etc/zfs/zed} directory and filling it appropriately,
+then launching @code{zed}.
+
+@item
+@file{/etc/zfs/zpool.cache}. Currently the ZFS support on Guix always forces
+scanning of all devices at bootup to look for ZFS pools. For systems with
+dozens or hundreds of storage devices, this can lead to slow bootup. One issue
+is that tools should really not write to @code{/etc} which is supposed to be for
+configuration; possibly it could be moved to @code{/var} instead. Another issue
+is that if Guix ever supports @code{/} on ZFS, we would need to somehow keep the
+@code{zpool.cache} file inside the @code{initrd} up-to-date with what is in the
+@code{/} mount point.
+
+@item
+@code{zfs share}. This will require some (unknown amount of) work to integrate
+into the Samba and NFS services of Guix. You @emph{can} manually set up Samba
+and NFS to share any mounted ZFS datasets by setting up their configurations
+properly; it just can't be done for you by @code{zfs share} and the
+@code{sharesmb} and @code{sharenfs} properties.
+@end enumerate
+
+Hopefully, support for the above only requires code to be written, so users
+are encouraged to hack on Guix to implement the above features.
+
@node Mapped Devices
@section Mapped Devices
diff --git a/gnu/local.mk b/gnu/local.mk
index 3d4147a879..6fd8284a9b 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -40,6 +40,7 @@
# Copyright © 2020 Malte Frank Gerdes <mate.f.gerdes@gmail.com>
# Copyright © 2020 Vinicius Monego <monego@posteo.net>
# Copyright © 2021 Björn Höfling <bjoern.hoefling@bjoernhoefling.de>
+# Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
#
# This file is part of GNU Guix.
#
@@ -607,6 +608,7 @@ GNU_SYSTEM_MODULES = \
%D%/services/docker.scm \
%D%/services/authentication.scm \
%D%/services/file-sharing.scm \
+ %D%/services/file-systems.scm \
%D%/services/games.scm \
%D%/services/ganeti.scm \
%D%/services/getmail.scm \
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index f50bcfdcb4..09a2d7c45b 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -187,7 +187,9 @@
references-file
- %base-services))
+ %base-services
+
+ dependency->shepherd-service-name))
;;; Commentary:
;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..0b1aae38ac
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,295 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+ #:use-module (gnu packages file-systems)
+ #:use-module (gnu services)
+ #:use-module (gnu services base)
+ #:use-module (gnu services linux)
+ #:use-module (gnu services mcron)
+ #:use-module (gnu services shepherd)
+ #:use-module (gnu system mapped-devices)
+ #:use-module (guix gexp)
+ #:use-module (guix packages)
+ #:use-module (guix records)
+ #:export (zfs-service-type
+
+ zfs-configuration
+ zfs-configuration?
+ zfs-configuration-kernel
+ zfs-configuration-base-zfs
+ zfs-configuration-base-zfs-auto-snapshot
+ zfs-configuration-dependencies
+ zfs-configuration-auto-mount?
+ zfs-configuration-auto-scrub
+ zfs-configuration-auto-snapshot?
+ zfs-configuration-auto-snapshot-keep
+
+ %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+ zfs-configuration
+ make-zfs-configuration
+ zfs-configuration?
+
+ ; linux-libre kernel you want to compile the base-zfs module for.
+ (kernel zfs-configuration-kernel)
+
+ ; the OpenZFS package that will be modified to compile for the
+ ; given kernel.
+ (base-zfs zfs-configuration-base-zfs
+ (default zfs))
+
+ ; the zfs-auto-snapshot package that will be modified to compile
+ ; for the given kernel.
+ (base-zfs-auto-snapshot zfs-configuration-base-zfs-auto-snapshot
+ (default zfs-auto-snapshot))
+
+ ; list of <mapped-device> or <file-system> objects that must be
+ ; opened/mounted before we import any ZFS pools.
+ (dependencies zfs-configuration-dependencies
+ (default '()))
+
+ ; #t if mountable datasets are to be mounted automatically.
+ ; #f if not mounting.
+ ; #t is the expected behavior on other operating systems, the
+ ; #f is only supported for "rescue" operating systems where
+ ; the user wants lower-level control of when to mount.
+ (auto-mount? zfs-configuration-auto-mount?
+ (default #t))
+
+ ; 'weekly for weekly scrubbing, 'monthly for monthly scrubbing, an
+ ; mcron time specification that can be given to `job`, or #f to
+ ; disable.
+ (auto-scrub zfs-configuration-auto-scrub
+ (default 'weekly))
+
+ ; #t if auto-snapshot is default (and `com.sun:auto-snapshot=false`
+ ; disables auto-snapshot per dataset), #f if no auto-snapshotting
+ ; is default (and `com.sun:auto-snapshot=true` enables auto-snapshot
+ ; per dataset).
+ (auto-snapshot? zfs-configuration-auto-snapshot?
+ (default #t))
+
+ ; association list of symbol-number pairs to indicate the number
+ ; of automatic snapshots to keep for each of 'frequent, 'hourly,
+ ; 'daily, 'weekly, and 'monthly.
+ ; e.g. '((frequent . 8) (hourly . 12))
+ (auto-snapshot-keep zfs-configuration-auto-snapshot-keep
+ (default '())))
+
+(define %default-auto-snapshot-keep
+ '((frequent . 4)
+ (hourly . 24)
+ (daily . 31)
+ (weekly . 8)
+ (monthly . 12)))
+
+(define %auto-snapshot-mcron-schedule
+ '((frequent . "0,15,30,45 * * * *")
+ (hourly . "0 * * * *")
+ (daily . "0 0 * * *")
+ (weekly . "0 0 * * 7")
+ (monthly . "0 0 1 * *")))
+
+;; A synthetic and unusable MAPPED-DEVICE intended for use when
+;; the user has created a mountable filesystem inside a ZFS
+;; zvol and wants it mounted inside the configuration.scm.
+(define %zfs-zvol-dependency
+ (mapped-device
+ (source '())
+ (targets '("zvol/*"))
+ (type #f)))
+
+(define (make-zfs-package conf)
+ (let ((kernel (zfs-configuration-kernel conf))
+ (base-zfs (zfs-configuration-base-zfs conf)))
+ (package
+ (inherit base-zfs)
+ (arguments (cons* #:linux kernel
+ (package-arguments base-zfs))))))
+
+(define (make-zfs-auto-snapshot-package conf)
+ (let ((zfs (make-zfs-package conf))
+ (base-zfs-auto-snapshot (zfs-configuration-base-zfs-auto-snapshot conf)))
+ (package
+ (inherit base-zfs-auto-snapshot)
+ (inputs `(("zfs" ,zfs))))))
+
+(define (zfs-loadable-modules conf)
+ (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (zfs (file-append zfs-package "/sbin/zfs"))
+ (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
+ (scheme-modules `((srfi srfi-1)
+ (srfi srfi-34)
+ (srfi srfi-35)
+ (rnrs io ports)
+ ,@%default-modules)))
+ (define zfs-scan
+ (shepherd-service
+ (provision '(zfs-scan))
+ (requirement `(root-file-system
+ kernel-module-loader
+ udev
+ ,@(map dependency->shepherd-service-name
+ (zfs-configuration-dependencies conf))))
+ (documentation "Scans for and imports ZFS pools.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error importing pools: ~s~%"
+ (condition-message c))
+ #f))
+ ; TODO: optionally use a cachefile.
+ (invoke #$zpool "import" "-a" "-N"))))
+ ;; Why not one-shot? Because we don't really want to rescan
+ ;; this each time a requiring process is restarted, as scanning
+ ;; can take a long time and a lot of I/O.
+ (stop #~(const #f))))
+
+ (define device-mapping-zvol/*
+ (shepherd-service
+ (provision '(device-mapping-zvol/*))
+ (requirement '(zfs-scan))
+ (documentation "Waits for all ZFS ZVOLs to be opened.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error opening zvols: ~s~%"
+ (condition-message c))
+ #f))
+ (invoke #$zvol_wait))))
+ (stop #~(const #f))))
+
+ (define zfs-auto-mount
+ (shepherd-service
+ (provision '(zfs-auto-mount))
+ (requirement '(zfs-scan))
+ (documentation "Mounts all non-legacy mounted ZFS filesystems.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error mounting file systems: ~s~%"
+ (condition-message c))
+ #f))
+ ;; Output to current-error-port, otherwise the
+ ;; user will not see any prompts for passwords
+ ;; of encrypted datasets.
+ ;; XXX Maybe better to explicitly open /dev/console ?
+ (with-output-to-port (current-error-port)
+ (lambda ()
+ (invoke #$zfs "mount" "-a" "-l"))))))
+ (stop #~(lambda _
+ ;; Make sure that Shepherd does not have a CWD that
+ ;; is a mounted ZFS filesystem, which would prevent
+ ;; unmounting.
+ (chdir "/")
+ (invoke #$zfs "unmount" "-a" "-f")))))
+
+ `(,zfs-scan
+ ,device-mapping-zvol/*
+ ,@(if (zfs-configuration-auto-mount? conf)
+ `(,zfs-auto-mount)
+ '()))))
+
+(define (zfs-user-processes conf)
+ (if (zfs-configuration-auto-mount? conf)
+ '(zfs-auto-mount)
+ '(zfs-scan)))
+
+(define (zfs-mcron-auto-snapshot-jobs conf)
+ (let* ((user-auto-snapshot-keep (zfs-configuration-auto-snapshot-keep conf))
+ ;; assoc-ref has earlier entries overriding later ones.
+ (auto-snapshot-keep (append user-auto-snapshot-keep
+ %default-auto-snapshot-keep))
+ (auto-snapshot? (zfs-configuration-auto-snapshot? conf))
+ (zfs-auto-snapshot-package (make-zfs-auto-snapshot-package conf))
+ (zfs-auto-snapshot (file-append zfs-auto-snapshot-package
+ "/sbin/zfs-auto-snapshot")))
+ (map
+ (lambda (label)
+ (let ((keep (assoc-ref auto-snapshot-keep label))
+ (sched (assoc-ref %auto-snapshot-mcron-schedule label)))
+ #~(job '#$sched
+ (string-append #$zfs-auto-snapshot
+ " --quiet --syslog "
+ " --label=" #$(symbol->string label)
+ " --keep=" #$(number->string keep)
+ " //"))))
+ '(frequent hourly daily weekly monthly))))
+
+(define (zfs-mcron-auto-scrub-jobs conf)
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (auto-scrub (zfs-configuration-auto-scrub conf))
+ (sched (cond
+ ((eq? auto-scrub 'weekly) "0 0 * * 7")
+ ((eq? auto-scrub 'monthly) "0 0 1 * *")
+ (else auto-scrub))))
+ (list
+ #~(job '#$sched
+ ;; Suppress errors: if there are no ZFS pools, the
+ ;; scrub will not be given any arguments, which makes
+ ;; it error out.
+ (string-append "(" #$zpool " scrub `" #$zpool " list -o name -H` "
+ "> /dev/null 2>&1) "
+ "|| exit 0")))))
+
+(define (zfs-mcron-jobs conf)
+ (append (zfs-mcron-auto-snapshot-jobs conf)
+ (if (zfs-configuration-auto-scrub conf)
+ (zfs-mcron-auto-scrub-jobs conf)
+ '())))
+
+(define zfs-service-type
+ (service-type
+ (name 'zfs)
+ (extensions
+ (list ;; Install OpenZFS kernel module into kernel profile.
+ (service-extension linux-loadable-module-service-type
+ zfs-loadable-modules)
+ ;; And load it.
+ (service-extension kernel-module-loader-service-type
+ (const '("zfs")))
+ ;; Make sure ZFS pools and datasets are mounted at
+ ;; boot.
+ (service-extension shepherd-root-service-type
+ zfs-shepherd-services)
+ ;; Make sure user-processes don't start until
+ ;; after ZFS does.
+ (service-extension user-processes-service-type
+ zfs-user-processes)
+ ;; Install automated scrubbing and snapshotting.
+ (service-extension mcron-service-type
+ zfs-mcron-jobs)
+
+ ;; Install ZFS management commands in the system
+ ;; profile.
+ (service-extension profile-service-type
+ (compose list make-zfs-package))
+ ;; Install ZFS udev rules.
+ (service-extension udev-service-type
+ (compose list make-zfs-package))))
+ (description "Installs ZFS, an advanced filesystem and volume manager.")))
--
2.31.0
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 0/3] New patch series for Even Better ZFS Support on Guix
2021-03-22 14:33 ` [bug#45692] [PATCH v3 0/3] New patch series for " raid5atemyhomework via Guix-patches via
@ 2021-03-28 12:55 ` Léo Le Bouter via Guix-patches via
2021-03-29 4:39 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: Léo Le Bouter via Guix-patches via @ 2021-03-28 12:55 UTC (permalink / raw)
To: raid5atemyhomework, 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 1485 bytes --]
Thanks a lot for the patches!
On Mon, 2021-03-22 at 14:33 +0000, raid5atemyhomework via Guix-patches
via wrote:
> Personally I'm not particularly motivated to push this anymore. I
> have a frankenstein's `configuration.scm` that gets me the bits of
> ZFS functionality that I absolutely need, with the rest something I'm
> just living without, and while it would be *nice* to have *some* out-
> of-the-box support for ZFS (or at least make it easy to get a box to
> get support for ZFS out of), I personally don't need it at this
> point.
I understand your frustration but also understand that the people who
are able to review your patches can't make themselves available in the
way you desire, there's lots of pending patches to GNU Guix and as much
as ZFS support is valuable, there's also lots of other things that are
valuable and are in the backlog.
> Patch 1/3 corresponds to the original Patch 1/4. Patch 2/3 has no
> corresponding patch in the original patch set, but is the same patch
> as https://issues.guix.gnu.org/47134 which nobody is looking at, as
> usual. Patch 3/3 corresponds to the original Patch 4/4.
Please do not blame or shame us for not looking at patches, it's not
very nice, we do our best, at all times. We are overworked and the
people who are knowledgeable about GNU Guix and Scheme particularily
so. I don't feel like I am knowledgeable about Scheme and GNU Guix
enough to review your patches, sorry for that.
Léo
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 0/3] New patch series for Even Better ZFS Support on Guix
2021-03-28 12:55 ` Léo Le Bouter via Guix-patches via
@ 2021-03-29 4:39 ` raid5atemyhomework via Guix-patches via
2021-07-23 15:11 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-03-29 4:39 UTC (permalink / raw)
To: Léo Le Bouter; +Cc: 45692@debbugs.gnu.org
> On Mon, 2021-03-22 at 14:33 +0000, raid5atemyhomework via Guix-patches
> via wrote:
>
> > Personally I'm not particularly motivated to push this anymore. I
> > have a frankenstein's `configuration.scm` that gets me the bits of
> > ZFS functionality that I absolutely need, with the rest something I'm
> > just living without, and while it would be nice to have some out-
> > of-the-box support for ZFS (or at least make it easy to get a box to
> > get support for ZFS out of), I personally don't need it at this
> > point.
>
> I understand your frustration but also understand that the people who
> are able to review your patches can't make themselves available in the
> way you desire, there's lots of pending patches to GNU Guix and as much
> as ZFS support is valuable, there's also lots of other things that are
> valuable and are in the backlog.
>
> > Patch 1/3 corresponds to the original Patch 1/4. Patch 2/3 has no
> > corresponding patch in the original patch set, but is the same patch
> > as https://issues.guix.gnu.org/47134 which nobody is looking at, as
> > usual. Patch 3/3 corresponds to the original Patch 4/4.
>
> Please do not blame or shame us for not looking at patches, it's not
> very nice, we do our best, at all times. We are overworked and the
> people who are knowledgeable about GNU Guix and Scheme particularily
> so. I don't feel like I am knowledgeable about Scheme and GNU Guix
> enough to review your patches, sorry for that.
Okay.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 2/3] gnu: Add zfs-auto-snapshot.
2021-03-22 14:34 ` [bug#45692] [PATCH v3 2/3] gnu: Add zfs-auto-snapshot raid5atemyhomework via Guix-patches via
@ 2021-05-11 14:05 ` Danny Milosavljevic
2021-05-13 1:21 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: Danny Milosavljevic @ 2021-05-11 14:05 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 45692@debbugs.gnu.org
Hi,
> + (invoke "make" "install"
> + "PREFIX="
> + (string-append "DESTDIR=" out)))
Are you sure about that?
Usually, DESTDIR is in order to supply a temporary build root (which will not
be referred-to by the installed programs--because that directory will be
deleted after the build) and PREFIX is in order to supply an actual
installation directory (installed programs can refer to that whenever they
want).
Is it different for this program?
Also, I'd suggest to use #:make-flags so other phases can also see the PREFIX
chosen. Otherwise, those could erroneously default to another prefix (for
loading of config files at runtime etc).
What do you think?
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 1/3] gnu: Allow services to install kernel-loadable modules.
2021-03-22 14:33 ` [bug#45692] [PATCH v3 1/3] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
@ 2021-05-11 14:17 ` Danny Milosavljevic
0 siblings, 0 replies; 81+ messages in thread
From: Danny Milosavljevic @ 2021-05-11 14:17 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 86 bytes --]
Pushed this patch to guix master as commit a3df382525ac515d1aa083f7f5bd3bd31eb1df40.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 2/3] gnu: Add zfs-auto-snapshot.
2021-05-11 14:05 ` Danny Milosavljevic
@ 2021-05-13 1:21 ` raid5atemyhomework via Guix-patches via
2021-05-13 13:08 ` Danny Milosavljevic
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-05-13 1:21 UTC (permalink / raw)
To: Danny Milosavljevic; +Cc: 45692@debbugs.gnu.org
Hello Danny,
> Hi,
>
> > - (invoke "make" "install"
> >
> >
> > - "PREFIX="
> >
> >
> > - (string-append "DESTDIR=" out)))
> >
> >
>
> Are you sure about that?
>
> Usually, DESTDIR is in order to supply a temporary build root (which will not
> be referred-to by the installed programs--because that directory will be
> deleted after the build) and PREFIX is in order to supply an actual
> installation directory (installed programs can refer to that whenever they
> want).
>
> Is it different for this program?
>
> Also, I'd suggest to use #:make-flags so other phases can also see the PREFIX
> chosen. Otherwise, those could erroneously default to another prefix (for
> loading of config files at runtime etc).
>
> What do you think?
Yes, I am quite sure; the program uses a custom `Makefile` that is fairly simple and can be trivially quoted here for your review:
```Makefile
PREFIX := /usr/local
all:
install:
install -d $(DESTDIR)/etc/cron.d
install -d $(DESTDIR)/etc/cron.daily
install -d $(DESTDIR)/etc/cron.hourly
install -d $(DESTDIR)/etc/cron.weekly
install -d $(DESTDIR)/etc/cron.monthly
install -m 0644 etc/zfs-auto-snapshot.cron.frequent $(DESTDIR)/etc/cron.d/zfs-auto-snapshot
install etc/zfs-auto-snapshot.cron.hourly $(DESTDIR)/etc/cron.hourly/zfs-auto-snapshot
install etc/zfs-auto-snapshot.cron.daily $(DESTDIR)/etc/cron.daily/zfs-auto-snapshot
install etc/zfs-auto-snapshot.cron.weekly $(DESTDIR)/etc/cron.weekly/zfs-auto-snapshot
install etc/zfs-auto-snapshot.cron.monthly $(DESTDIR)/etc/cron.monthly/zfs-auto-snapshot
install -d $(DESTDIR)$(PREFIX)/share/man/man8
install -m 0644 src/zfs-auto-snapshot.8 $(DESTDIR)$(PREFIX)/share/man/man8/zfs-auto-snapshot.8
install -d $(DESTDIR)$(PREFIX)/sbin
install src/zfs-auto-snapshot.sh $(DESTDIR)$(PREFIX)/sbin/zfs-auto-snapshot
uninstall:
rm $(DESTDIR)/etc/cron.d/zfs-auto-snapshot
rm $(DESTDIR)/etc/cron.hourly/zfs-auto-snapshot
rm $(DESTDIR)/etc/cron.daily/zfs-auto-snapshot
rm $(DESTDIR)/etc/cron.weekly/zfs-auto-snapshot
rm $(DESTDIR)/etc/cron.monthly/zfs-auto-snapshot
rm $(DESTDIR)$(PREFIX)/share/man/man8/zfs-auto-snapshot.8
rm $(DESTDIR)$(PREFIX)/sbin/zfs-auto-snapshot
```
Notice how it does not modify any of the source files at all, so the sources do not refer to anything provided in `$PREFIX`. However, it does create directories in `$DESTDIR`, so it has to use `$DESTDIR`.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 2/3] gnu: Add zfs-auto-snapshot.
2021-05-13 1:21 ` raid5atemyhomework via Guix-patches via
@ 2021-05-13 13:08 ` Danny Milosavljevic
0 siblings, 0 replies; 81+ messages in thread
From: Danny Milosavljevic @ 2021-05-13 13:08 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 131 bytes --]
Hi,
thanks for the explanation!
Pushed zfs-auto-snapshot to guix master as commit ba3b295a3ee956ac7500b5f9bb1d151b28ab30ed.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 0/3] New patch series for Even Better ZFS Support on Guix
2021-03-29 4:39 ` raid5atemyhomework via Guix-patches via
@ 2021-07-23 15:11 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-07-23 15:11 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
Can I get any particular feedback here at this point, from anyone at all?
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v3 3/3] gnu: Add ZFS service type.
2021-03-22 14:35 ` [bug#45692] [PATCH v3 3/3] gnu: Add ZFS service type raid5atemyhomework via Guix-patches via
@ 2021-07-25 14:03 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-07-25 14:03 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
Because it has been so long, the latest patch has bitrotted.
I really think at this point shaming people into reviewing this is my only option.
Give me a few hours or days to get a new patch that applies cleanly to origin/master.
Thanks
raid5atemyhomewkr
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (7 preceding siblings ...)
2021-03-22 14:35 ` [bug#45692] [PATCH v3 3/3] gnu: Add ZFS service type raid5atemyhomework via Guix-patches via
@ 2021-07-25 14:31 ` raid5atemyhomework via Guix-patches via
2021-08-01 9:41 ` raid5atemyhomework via Guix-patches via
` (2 more replies)
2021-09-02 21:24 ` [bug#45692] Gaslighting Mason Loring Bliss
` (3 subsequent siblings)
12 siblings, 3 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-07-25 14:31 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
Hello nonexistent reviewer,
I updated this patch to latest `origin/master` because the previous version has bitrotted and will not `git am` cleanly anymore.
There are no changes relative to v3, just rebased it so that the patch applies cleanly.
Testing has been very minimal: I created a VM with the added service, then ran it in a VM session that included additional devices (qcow2 files) from a previous VM run that formatted those as devices of a ZFS pool, and confirmed that the new VM could read it and manage that pool.
Is there any chance this will get reviewed or should I just not bother and move on with my life and forget about Guix?
At this point as well, I would like to point out what I think is a failing of how the Guix distribution is managed.
Guix does not assign any particular committers to any particular tasks or areas.
The intent is that any committer can review and commit any particular patch.
However, this fails as the Guix project has grown.
No single committer wants to review ***all*** the available patches, and this is understandable, as the Guix project has grown significantly and includes a wide variety of people with diverging interests and as an open-source project you cannot really require that particular people look at particular things.
Unfortunately, I do not know *who* committers are, and more importantly, *which* committer might be interested in this ZFS service type.
Because "any committer can review and commit any patch!!" there is no particular list or table I can refer to, to figure out who might be useful to ping for this patchset.
At the same time, because no committer is interested in *all* patches I cannot just ping some particular person and expect to somehow get on some list somewhere that tells me "you will be the 48486th patch that will be reviewed by <foo> who is interested in all patches".
It is very discouraging to work on this code for a few weeks, release it, not get any reviews, and end up in a situation where I have to make annoying small changes just to keep the patch from bitrotting.
I understand that there are few possible reviewers, but if potential new contributors get discouraged from contributing because they do not see their code actually getting in, then you really cannot expect the number of reviewers to increase, either.
I think it would be nice if I could at least be told some number of people who *might* be interested in this patch, or just throw the towel and not bother.
Thanks
raid5atemyhomework
From 5351aa7c1c14d4fea032adad895c436e02d1f261 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Mon, 22 Mar 2021 16:26:28 +0800
Subject: [PATCH] gnu: Add ZFS service type.
* gnu/services/file-systems.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
* gnu/services/base.scm: Export dependency->shepherd-service-name.
* doc/guix.texi (ZFS File System): New subsection.
---
doc/guix.texi | 351 ++++++++++++++++++++++++++++++++++
gnu/local.mk | 2 +
gnu/services/base.scm | 4 +-
gnu/services/file-systems.scm | 295 ++++++++++++++++++++++++++++
4 files changed, 651 insertions(+), 1 deletion(-)
create mode 100644 gnu/services/file-systems.scm
diff --git a/doc/guix.texi b/doc/guix.texi
index b3c16e6507..e21c47d7ca 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@*
Copyright @copyright{} 2021 Raghav Gururajan@*
Copyright @copyright{} 2021 Domagoj Stolfa@*
Copyright @copyright{} 2021 Hui Lu@*
+Copyright @copyright{} 2021 raid5atemyhomework@*
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -14265,6 +14266,356 @@ a file system declaration such as:
compress-force=zstd,space_cache=v2"))
@end lisp
+
+@node ZFS File System
+@subsection ZFS File System
+
+Support for ZFS file systems is provided on Guix by the OpenZFS project.
+OpenZFS currently only supports Linux-Libre and is not available on the
+Hurd.
+
+OpenZFS is free software; unfortunately its license is incompatible with
+the GNU General Public License (GPL), the license of the Linux kernel,
+which means they cannot be distributed together. However, as a user,
+you can choose to build ZFS and use it together with Linux; you can
+even rely on Guix to automate this task. See
+@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
+the Free Software Foundation} for more information.
+
+As a large and complex kernel module, OpenZFS has to be compiled for a
+specific version of Linux-Libre. At times, the latest OpenZFS package
+available in Guix is not compatible with the latest Linux-Libre version.
+Thus, directly installing the @code{zfs} package can fail.
+
+Instead, you are recommended to select a specific older long-term-support
+Linux-Libre kernel. Do not use @code{linux-libre-lts}, as even the
+latest long-term-support kernel may be too new for @code{zfs}. Instead,
+explicitly select a specific older version, such as @code{linux-libre-5.10},
+and upgrade it manually later as new long-term-support kernels become
+available that you have confirmed is compatible with the latest available
+OpenZFS version on Guix.
+
+For example, you can modify your system configuration file to a specific
+Linux-Libre version and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+ #;@dots{}
+ linux)
+(use-service-modules
+ #;@dots{}
+ file-systems)
+
+(define my-kernel linux-libre-5.10)
+
+(operating-system
+ (kernel my-kernel)
+ #;@dots{}
+ (services
+ (cons* (service zfs-service-type
+ (zfs-configuration
+ (kernel my-kernel)))
+ #;@dots{}
+ %desktop-services))
+ #;@dots{})
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type for a service that adds ZFS support to your operating
+system. The service is configured using a @code{zfs-configuration}
+record.
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+ (zfs-configuration
+ (kernel linux-libre-5.4)))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS support in Guix
+System. Its fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux-Libre kernel to compile OpenZFS for. This field
+is always required. It @emph{must} be the same kernel you use in your
+@code{operating-system} form.
+
+@item @code{base-zfs} (default: @code{zfs})
+The OpenZFS package that will be compiled for the given Linux-Libre kernel.
+
+@item @code{base-zfs-auto-snapshot} (default: @code{zfs-auto-snapshot})
+The @code{zfs-auto-snapshot} package to use. It will be modified to
+specifically use the OpenZFS compiled for your kernel.
+
+@item @code{dependencies} (default: @code{'()})
+A list of @code{<file-system>} or @code{<mapped-device>} records that must
+be mounted or opened before OpenZFS scans for pools to import. For example,
+if you have set up LUKS containers as leaf VDEVs in a pool, you have to
+include their corresponding @code{<mapped-ddevice>} records so that OpenZFS
+can import the pool correctly at bootup.
+
+@item @code{auto-mount?} (default: @code{#t})
+Whether to mount datasets with the ZFS @code{mountpoint} property automatically
+at startup. This is the behavior that ZFS users usually expect. You might
+set this to @code{#f} for an operating system intended as a ``rescue'' system
+that is intended to help debug problems with the disks rather than actually
+work in production.
+
+@item @code{auto-scrub} (default: @code{'weekly})
+Specifies how often to scrub all pools. Can be the symbols @code{'weekly} or
+@code{'monthly}, or a schedule specification understood by
+@xref{mcron, mcron job specifications,, mcron, GNU@tie{}mcron}, such as
+@code{"0 3 * * 6"} for ``every 3AM on Saturday''.
+It can also be @code{#f} to disable auto-scrubbing (@strong{not recommended}).
+
+The general guideline is to scrub weekly when using consumer-quality drives, and
+to scrub monthly when using enterprise-quality drives.
+
+@code{'weekly} scrubs are done on Sunday midnight, while @code{monthly} scrubs
+are done on midnight on the first day of each month.
+
+@item @code{auto-snapshot?} (default: @code{#t})
+Specifies whether to auto-snapshot by default. If @code{#t}, then snapshots
+are automatically created except for ZFS datasets with the
+@code{com.sun:auto-snapshot} ZFS vendor property set to @code{false}.
+
+If @code{#f}, snapshots will not be automatically created, unless the ZFS
+dataset has the @code{com.sun:auto-snapshot} ZFS vendor property set to
+@code{true}.
+
+@item @code{auto-snapshot-keep} (default: @code{'()})
+Specifies an association list of symbol-number pairs, indicating the number
+of automatically-created snapshots to retain for each frequency type.
+
+If not specified via this field, by default there are 4 @code{frequent}, 24
+@code{hourly}, 31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly} snapshots.
+
+For example:
+
+@lisp
+(zfs-configuration
+ (kernel my-kernel)
+ (auto-snapshot-keep
+ '((frequent . 8)
+ (hourly . 12))))
+@end lisp
+
+The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
+@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
+defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
+
+@end table
+@end deftp
+
+@subsubsection ZFS Auto-Snapshot
+
+The ZFS service on Guix System supports auto-snapshots as implemented in the
+Solaris operating system.
+
+@code{frequent} (every 15 minutes), @code{hourly}, @code{daily}, @code{weekly},
+and @code{monthly} snapshots are created automatically for ZFS datasets that
+have auto-snapshot enabled. They will be named, for example,
+@code{zfs-auto-snap_frequent-2021-03-22-1415}. You can continue to use
+manually-created snapshots as long as they do not conflict with the naming
+convention used by auto-snapshot. You can also safely manually destroy
+automatically-created snapshots, for example to free up space.
+
+The @code{com.sun:auto-snapshot} ZFS property controls auto-snapshot on a
+per-dataset level. Sub-datasets will inherit this property from their parent
+dataset, but can have their own property.
+
+You @emph{must} set this property to @code{true} or @code{false} exactly,
+otherwise it will be treated as if the property is unset.
+
+For example:
+
+@example
+# zfs list -o name
+NAME
+tank
+tank/important-data
+tank/tmp
+# zfs set com.sun:auto-snapshot=true tank
+# zfs set com.sun:auto-snapshot=false tank/tmp
+@end example
+
+The above will set @code{tank} and @code{tank/important-data} to be
+auto-snapshot, while @code{tank/tmp} will not be auto-snapshot.
+
+If the @code{com.sun:auto-snapshot} property is not set for a dataset
+(the default when pools and datasets are created), then whether
+auto-snapshot is done or not will depend on the @code{auto-snapshot?}
+field of the @code{zfs-configuration} record.
+
+There are also @code{com.sun:auto-snapshot:frequent},
+@code{com.sun:auto-snapshot:hourly}, @code{com.sun:auto-snapshot:daily},
+@code{com.sun:auto-snapshot:weekly}, and @code{com.sun:auto-snapshot:monthly}
+properties that give finer-grained control of whether to auto-snapshot a
+dataset at a particular schedule.
+
+The number of snapshots kept for all datasets can be overridden via the
+@code{auto-snapshot-keep} field of the @code{zfs-configuration} record.
+There is currently no support to have different numbers of snapshots to
+keep for different datasets.
+
+@subsubsection ZVOLs
+
+ZFS supports ZVOLs, block devices that ZFS exposes to the operating
+system in the @code{/dev/zvol/} directory. The ZVOL will have the same
+resilience and self-healing properties as other datasets on your ZFS pool.
+ZVOLs can also be snapshotted (and will be included in auto-snapshotting
+if enabled), which snapshots the state of the block device, effectively
+snapshotting the hosted file system.
+
+You can put any file system inside the ZVOL. However, in order to mount this
+file system at system start, you need to add @code{%zfs-zvol-dependency} as a
+dependency of each file system inside a ZVOL.
+
+@defvr {Scheme Variable} %zfs-zvol-dependency
+An artificial @code{<mapped-device>} which tells the file system mounting
+service to wait for ZFS to provide ZVOLs before mounting the
+@code{<file-system>} dependent on it.
+@end defvr
+
+For example, suppose you create a ZVOL and put an ext4 filesystem
+inside it:
+
+@example
+# zfs create -V 100G tank/ext4-on-zfs
+# mkfs.ext4 /dev/zvol/tank/ext4-on-zfs
+# mkdir /ext4-on-zfs
+# mount /dev/zvol/tank/ext4-on-zfs /ext4-on-zfs
+@end example
+
+You can then set this up to be mounted at boot by adding this to the
+@code{file-systems} field of your @code{operating-system} record:
+
+@lisp
+(file-system
+ (device "/dev/zvol/tank/ext4-on-zfs")
+ (mount-point "/ext4-on-zfs")
+ (type "ext4")
+ (dependencies (list %zfs-zvol-dependency)))
+@end lisp
+
+You @emph{must not} add @code{%zfs-zvol-dependency} to your
+@code{operating-system}'s @code{mapped-devices} field, and you @emph{must
+not} add it (or any @code{<file-system>}s dependent on it) to the
+@code{dependencies} field of @code{zfs-configuration}. Finally, you
+@emph{must not} use @code{%zfs-zvol-dependency} unless you actually
+instantiate @code{zfs-service-type} on your system.
+
+@subsubsection Unsupported Features
+
+Some common features and uses of ZFS are currently not supported, or not
+fully supported, on Guix.
+
+@enumerate
+@item
+Shepherd-managed daemons that are configured to read from or write to ZFS
+mountpoints need to include @code{user-processes} in their @code{requirement}
+field. This is the earliest that ZFS file systems are assured of being
+mounted.
+
+Generally, most daemons will, directly or indirectly, require
+@code{networking}, or @code{user-processes}, or both. Most implementations
+of @code{networking} will also require @code{user-processes} so daemons that
+require only @code{networking} will also generally start up after
+@code{user-processes}. A notable exception, however, is
+@code{static-networking-service-type}. You will need to explicitly add
+@code{user-processes} as a @code{requirement} of your @code{static-networking}
+record.
+
+@item
+@code{mountpoint=legacy} ZFS file systems. The handlers for the Guix mounting
+system have not yet been modified to support ZFS, and will expect @code{/dev}
+paths in the @code{<file-system>}'s @code{device} field, but ZFS file systems
+are referred to via non-path @code{pool/file/system} names. Such file systems
+also need to be mounted @emph{after} OpenZFS has scanned for pools.
+
+You can still manually mount these file systems after system boot; what is
+only unsupported is mounting them automatically at system boot by specifying
+them in @code{<file-system>} records of your @code{operating-system}.
+
+@item
+@code{/home} on ZFS. Guix will create home directories for users, but this
+process currently cannot be scheduled after ZFS file systems are mounted.
+Thus, the ZFS file system might be mounted @emph{after} Guix has created
+home directories at boot, at which point OpenZFS will refuse to mount since
+the mountpoint is not empty. However, you @emph{can} create an ext4, xfs,
+btrfs, or other supported file system inside a ZVOL, have that depend on
+@code{%zfs-zvol-dependency}, and set it to mount on the @code{/home}
+directory; they will be scheduled to mount before the @code{user-homes}
+process.
+
+Similarly, other locations like @code{/var}, @code{/gnu/store} and so
+on cannot be reliably put in a ZFS file system, though they may be
+possible to create as other file systems inside ZVOL containers.
+
+@item
+@code{/} and @code{/boot} on ZFS. These require Guix to expose more of
+the @code{initrd} very early boot process to services. It also requires
+Guix to have the ability to explicitly load modules while still in
+@code{initrd} (currently kernel modules loaded by
+@code{kernel-module-loader-service-type} are loaded after @code{/} is
+mounted). Further, since one of ZFS's main advantages is that it can
+continue working despite the loss of one or more devices, it makes sense
+to also support installing the bootloader on all devices of the pool that
+contains the @code{/} and @code{/boot}; after all, if ZFS can survive the
+loss of one device, the bootloader should also be able to survive the loss
+of one device.
+
+@item
+ZVOL swap devices. Mapped swap devices need to be listed in
+@code{mapped-devices} to ensure they are opened before the system attempts
+to use them, but you cannot currently add @code{%zfs-zvol-dependency} to
+@code{mapped-devices}.
+
+This will also require significant amounts of testing, as various kernel
+build options and patches may affect how swapping works, which are possibly
+different on Guix System compared to other distributions that this feature is
+known to work on.
+
+@item
+ZFS Event Daemon. Support for this has not been written yet, patches are
+welcome. The main issue is how to design this in a Guix style while
+supporting legacy shell-script styles as well. In particular, OpenZFS itself
+comes with a number of shell scripts intended for ZFS Event Daemon, and we
+need to figure out how the user can choose to use or not use the provided
+scripts (and configure any settings they have) or override with their own
+custom code (which could be shell scripts they have written and trusted from
+previous ZFS installations).
+
+As-is, you can create your own service that activates the ZFS Event Daemon
+by creating the @file{/etc/zfs/zed} directory and filling it appropriately,
+then launching @code{zed}.
+
+@item
+@file{/etc/zfs/zpool.cache}. Currently the ZFS support on Guix always forces
+scanning of all devices at bootup to look for ZFS pools. For systems with
+dozens or hundreds of storage devices, this can lead to slow bootup. One issue
+is that tools should really not write to @code{/etc} which is supposed to be for
+configuration; possibly it could be moved to @code{/var} instead. Another issue
+is that if Guix ever supports @code{/} on ZFS, we would need to somehow keep the
+@code{zpool.cache} file inside the @code{initrd} up-to-date with what is in the
+@code{/} mount point.
+
+@item
+@code{zfs share}. This will require some (unknown amount of) work to integrate
+into the Samba and NFS services of Guix. You @emph{can} manually set up Samba
+and NFS to share any mounted ZFS datasets by setting up their configurations
+properly; it just can't be done for you by @code{zfs share} and the
+@code{sharesmb} and @code{sharenfs} properties.
+@end enumerate
+
+Hopefully, support for the above only requires code to be written, o users
+are encouraged to hack on Guix to implement the above features.
+
@node Mapped Devices
@section Mapped Devices
diff --git a/gnu/local.mk b/gnu/local.mk
index b944c671af..a2ff871277 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -43,6 +43,7 @@
# Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
# Copyright © 2021 Arun Isaac <arunisaac@systemreboot.net>
# Copyright © 2021 Sharlatan Hellseher <sharlatanus@gmail.com>
+# Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
#
# This file is part of GNU Guix.
#
@@ -618,6 +619,7 @@ GNU_SYSTEM_MODULES = \
%D%/services/docker.scm \
%D%/services/authentication.scm \
%D%/services/file-sharing.scm \
+ %D%/services/file-systems.scm \
%D%/services/games.scm \
%D%/services/ganeti.scm \
%D%/services/getmail.scm \
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index ab3e441a7b..bcca24f93a 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -185,7 +185,9 @@
references-file
- %base-services))
+ %base-services
+
+ dependency->shepherd-service-name))
;;; Commentary:
;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..0b1aae38ac
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,295 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+ #:use-module (gnu packages file-systems)
+ #:use-module (gnu services)
+ #:use-module (gnu services base)
+ #:use-module (gnu services linux)
+ #:use-module (gnu services mcron)
+ #:use-module (gnu services shepherd)
+ #:use-module (gnu system mapped-devices)
+ #:use-module (guix gexp)
+ #:use-module (guix packages)
+ #:use-module (guix records)
+ #:export (zfs-service-type
+
+ zfs-configuration
+ zfs-configuration?
+ zfs-configuration-kernel
+ zfs-configuration-base-zfs
+ zfs-configuration-base-zfs-auto-snapshot
+ zfs-configuration-dependencies
+ zfs-configuration-auto-mount?
+ zfs-configuration-auto-scrub
+ zfs-configuration-auto-snapshot?
+ zfs-configuration-auto-snapshot-keep
+
+ %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+ zfs-configuration
+ make-zfs-configuration
+ zfs-configuration?
+
+ ; linux-libre kernel you want to compile the base-zfs module for.
+ (kernel zfs-configuration-kernel)
+
+ ; the OpenZFS package that will be modified to compile for the
+ ; given kernel.
+ (base-zfs zfs-configuration-base-zfs
+ (default zfs))
+
+ ; the zfs-auto-snapshot package that will be modified to compile
+ ; for the given kernel.
+ (base-zfs-auto-snapshot zfs-configuration-base-zfs-auto-snapshot
+ (default zfs-auto-snapshot))
+
+ ; list of <mapped-device> or <file-system> objects that must be
+ ; opened/mounted before we import any ZFS pools.
+ (dependencies zfs-configuration-dependencies
+ (default '()))
+
+ ; #t if mountable datasets are to be mounted automatically.
+ ; #f if not mounting.
+ ; #t is the expected behavior on other operating systems, the
+ ; #f is only supported for "rescue" operating systems where
+ ; the user wants lower-level control of when to mount.
+ (auto-mount? zfs-configuration-auto-mount?
+ (default #t))
+
+ ; 'weekly for weekly scrubbing, 'monthly for monthly scrubbing, an
+ ; mcron time specification that can be given to `job`, or #f to
+ ; disable.
+ (auto-scrub zfs-configuration-auto-scrub
+ (default 'weekly))
+
+ ; #t if auto-snapshot is default (and `com.sun:auto-snapshot=false`
+ ; disables auto-snapshot per dataset), #f if no auto-snapshotting
+ ; is default (and `com.sun:auto-snapshot=true` enables auto-snapshot
+ ; per dataset).
+ (auto-snapshot? zfs-configuration-auto-snapshot?
+ (default #t))
+
+ ; association list of symbol-number pairs to indicate the number
+ ; of automatic snapshots to keep for each of 'frequent, 'hourly,
+ ; 'daily, 'weekly, and 'monthly.
+ ; e.g. '((frequent . 8) (hourly . 12))
+ (auto-snapshot-keep zfs-configuration-auto-snapshot-keep
+ (default '())))
+
+(define %default-auto-snapshot-keep
+ '((frequent . 4)
+ (hourly . 24)
+ (daily . 31)
+ (weekly . 8)
+ (monthly . 12)))
+
+(define %auto-snapshot-mcron-schedule
+ '((frequent . "0,15,30,45 * * * *")
+ (hourly . "0 * * * *")
+ (daily . "0 0 * * *")
+ (weekly . "0 0 * * 7")
+ (monthly . "0 0 1 * *")))
+
+;; A synthetic and unusable MAPPED-DEVICE intended for use when
+;; the user has created a mountable filesystem inside a ZFS
+;; zvol and wants it mounted inside the configuration.scm.
+(define %zfs-zvol-dependency
+ (mapped-device
+ (source '())
+ (targets '("zvol/*"))
+ (type #f)))
+
+(define (make-zfs-package conf)
+ (let ((kernel (zfs-configuration-kernel conf))
+ (base-zfs (zfs-configuration-base-zfs conf)))
+ (package
+ (inherit base-zfs)
+ (arguments (cons* #:linux kernel
+ (package-arguments base-zfs))))))
+
+(define (make-zfs-auto-snapshot-package conf)
+ (let ((zfs (make-zfs-package conf))
+ (base-zfs-auto-snapshot (zfs-configuration-base-zfs-auto-snapshot conf)))
+ (package
+ (inherit base-zfs-auto-snapshot)
+ (inputs `(("zfs" ,zfs))))))
+
+(define (zfs-loadable-modules conf)
+ (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (zfs (file-append zfs-package "/sbin/zfs"))
+ (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
+ (scheme-modules `((srfi srfi-1)
+ (srfi srfi-34)
+ (srfi srfi-35)
+ (rnrs io ports)
+ ,@%default-modules)))
+ (define zfs-scan
+ (shepherd-service
+ (provision '(zfs-scan))
+ (requirement `(root-file-system
+ kernel-module-loader
+ udev
+ ,@(map dependency->shepherd-service-name
+ (zfs-configuration-dependencies conf))))
+ (documentation "Scans for and imports ZFS pools.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error importing pools: ~s~%"
+ (condition-message c))
+ #f))
+ ; TODO: optionally use a cachefile.
+ (invoke #$zpool "import" "-a" "-N"))))
+ ;; Why not one-shot? Because we don't really want to rescan
+ ;; this each time a requiring process is restarted, as scanning
+ ;; can take a long time and a lot of I/O.
+ (stop #~(const #f))))
+
+ (define device-mapping-zvol/*
+ (shepherd-service
+ (provision '(device-mapping-zvol/*))
+ (requirement '(zfs-scan))
+ (documentation "Waits for all ZFS ZVOLs to be opened.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error opening zvols: ~s~%"
+ (condition-message c))
+ #f))
+ (invoke #$zvol_wait))))
+ (stop #~(const #f))))
+
+ (define zfs-auto-mount
+ (shepherd-service
+ (provision '(zfs-auto-mount))
+ (requirement '(zfs-scan))
+ (documentation "Mounts all non-legacy mounted ZFS filesystems.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error mounting file systems: ~s~%"
+ (condition-message c))
+ #f))
+ ;; Output to current-error-port, otherwise the
+ ;; user will not see any prompts for passwords
+ ;; of encrypted datasets.
+ ;; XXX Maybe better to explicitly open /dev/console ?
+ (with-output-to-port (current-error-port)
+ (lambda ()
+ (invoke #$zfs "mount" "-a" "-l"))))))
+ (stop #~(lambda _
+ ;; Make sure that Shepherd does not have a CWD that
+ ;; is a mounted ZFS filesystem, which would prevent
+ ;; unmounting.
+ (chdir "/")
+ (invoke #$zfs "unmount" "-a" "-f")))))
+
+ `(,zfs-scan
+ ,device-mapping-zvol/*
+ ,@(if (zfs-configuration-auto-mount? conf)
+ `(,zfs-auto-mount)
+ '()))))
+
+(define (zfs-user-processes conf)
+ (if (zfs-configuration-auto-mount? conf)
+ '(zfs-auto-mount)
+ '(zfs-scan)))
+
+(define (zfs-mcron-auto-snapshot-jobs conf)
+ (let* ((user-auto-snapshot-keep (zfs-configuration-auto-snapshot-keep conf))
+ ;; assoc-ref has earlier entries overriding later ones.
+ (auto-snapshot-keep (append user-auto-snapshot-keep
+ %default-auto-snapshot-keep))
+ (auto-snapshot? (zfs-configuration-auto-snapshot? conf))
+ (zfs-auto-snapshot-package (make-zfs-auto-snapshot-package conf))
+ (zfs-auto-snapshot (file-append zfs-auto-snapshot-package
+ "/sbin/zfs-auto-snapshot")))
+ (map
+ (lambda (label)
+ (let ((keep (assoc-ref auto-snapshot-keep label))
+ (sched (assoc-ref %auto-snapshot-mcron-schedule label)))
+ #~(job '#$sched
+ (string-append #$zfs-auto-snapshot
+ " --quiet --syslog "
+ " --label=" #$(symbol->string label)
+ " --keep=" #$(number->string keep)
+ " //"))))
+ '(frequent hourly daily weekly monthly))))
+
+(define (zfs-mcron-auto-scrub-jobs conf)
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (auto-scrub (zfs-configuration-auto-scrub conf))
+ (sched (cond
+ ((eq? auto-scrub 'weekly) "0 0 * * 7")
+ ((eq? auto-scrub 'monthly) "0 0 1 * *")
+ (else auto-scrub))))
+ (list
+ #~(job '#$sched
+ ;; Suppress errors: if there are no ZFS pools, the
+ ;; scrub will not be given any arguments, which makes
+ ;; it error out.
+ (string-append "(" #$zpool " scrub `" #$zpool " list -o name -H` "
+ "> /dev/null 2>&1) "
+ "|| exit 0")))))
+
+(define (zfs-mcron-jobs conf)
+ (append (zfs-mcron-auto-snapshot-jobs conf)
+ (if (zfs-configuration-auto-scrub conf)
+ (zfs-mcron-auto-scrub-jobs conf)
+ '())))
+
+(define zfs-service-type
+ (service-type
+ (name 'zfs)
+ (extensions
+ (list ;; Install OpenZFS kernel module into kernel profile.
+ (service-extension linux-loadable-module-service-type
+ zfs-loadable-modules)
+ ;; And load it.
+ (service-extension kernel-module-loader-service-type
+ (const '("zfs")))
+ ;; Make sure ZFS pools and datasets are mounted at
+ ;; boot.
+ (service-extension shepherd-root-service-type
+ zfs-shepherd-services)
+ ;; Make sure user-processes don't start until
+ ;; after ZFS does.
+ (service-extension user-processes-service-type
+ zfs-user-processes)
+ ;; Install automated scrubbing and snapshotting.
+ (service-extension mcron-service-type
+ zfs-mcron-jobs)
+
+ ;; Install ZFS management commands in the system
+ ;; profile.
+ (service-extension profile-service-type
+ (compose list make-zfs-package))
+ ;; Install ZFS udev rules.
+ (service-extension udev-service-type
+ (compose list make-zfs-package))))
+ (description "Installs ZFS, an advanced filesystem and volume manager.")))
--
2.31.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-07-25 14:31 ` [bug#45692] [PATCH v4 " raid5atemyhomework via Guix-patches via
@ 2021-08-01 9:41 ` raid5atemyhomework via Guix-patches via
2021-08-10 11:43 ` raid5atemyhomework via Guix-patches via
2021-09-02 20:57 ` Maxime Devos
2021-09-04 21:19 ` Xinglu Chen
2 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-08-01 9:41 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
BUMP
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-08-01 9:41 ` raid5atemyhomework via Guix-patches via
@ 2021-08-10 11:43 ` raid5atemyhomework via Guix-patches via
2021-08-31 0:48 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-08-10 11:43 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
BUMPITY BUMPITY BUMP BUMP BUMP
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-08-10 11:43 ` raid5atemyhomework via Guix-patches via
@ 2021-08-31 0:48 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-08-31 0:48 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
YOOOOOOOOOHOOOOOOOOOOOOOO
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-07-25 14:31 ` [bug#45692] [PATCH v4 " raid5atemyhomework via Guix-patches via
2021-08-01 9:41 ` raid5atemyhomework via Guix-patches via
@ 2021-09-02 20:57 ` Maxime Devos
2021-09-02 22:22 ` Maxime Devos
` (2 more replies)
2021-09-04 21:19 ` Xinglu Chen
2 siblings, 3 replies; 81+ messages in thread
From: Maxime Devos @ 2021-09-02 20:57 UTC (permalink / raw)
To: raid5atemyhomework, 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 36799 bytes --]
Hi,
Some comments on the code. Spoiler: the code is presumably good,
but there's a GPL violation.
> From 5351aa7c1c14d4fea032adad895c436e02d1f261 Mon Sep 17 00:00:00 2001
> From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
> Date: Mon, 22 Mar 2021 16:26:28 +0800
> Subject: [PATCH] gnu: Add ZFS service type.
>
> * gnu/services/file-systems.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
> * gnu/services/base.scm: Export dependency->shepherd-service-name.
> * doc/guix.texi (ZFS File System): New subsection.
> ---
> doc/guix.texi | 351 ++++++++++++++++++++++++++++++++++
> gnu/local.mk | 2 +
> gnu/services/base.scm | 4 +-
> gnu/services/file-systems.scm | 295 ++++++++++++++++++++++++++++
> 4 files changed, 651 insertions(+), 1 deletion(-)
> create mode 100644 gnu/services/file-systems.scm
>
> diff --git a/doc/guix.texi b/doc/guix.texi
> index b3c16e6507..e21c47d7ca 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@*
> Copyright @copyright{} 2021 Raghav Gururajan@*
> Copyright @copyright{} 2021 Domagoj Stolfa@*
> Copyright @copyright{} 2021 Hui Lu@*
> +Copyright @copyright{} 2021 raid5atemyhomework@*
>
> Permission is granted to copy, distribute and/or modify this document
> under the terms of the GNU Free Documentation License, Version 1.3 or
> @@ -14265,6 +14266,356 @@ a file system declaration such as:
> compress-force=zstd,space_cache=v2"))
> @end lisp
>
> +
> +@node ZFS File System
> +@subsection ZFS File System
> +
> +Support for ZFS file systems is provided on Guix by the OpenZFS project.
This is worded a bit misleading: it doesn't seem like OpenZFS is providing
Guix-specific support, except in the more general sense that OpenZFS provides a
ZFS kernel module and utilities to _everyone_. Suggestion:
‘Support for ZFS file systems in Guix is based on the OpenZFS project.’
> +OpenZFS currently only supports Linux-Libre and is not available on the
> +Hurd.
> +
> +OpenZFS is free software; unfortunately its license is incompatible with
> +the GNU General Public License (GPL), the license of the Linux kernel,
> +which means they cannot be distributed together. However, as a user,
> +you can choose to build ZFS and use it together with Linux; you can
> +even rely on Guix to automate this task. See
> +@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
> +the Free Software Foundation} for more information.
That analysis says explicitely that the CDDL is incompatible with the GPL,
and that they cannot be legally linked together. E.g., see the second quoted
paragraph:
‘A copyleft license, including any version of the GNU GPL or GNU AGPL, requires
augmented versions to be free under the same license -- the same requirement it
applies to modification of the code.1 I wrote a copyleft license for GNU programs
to ensure that all users of all versions of them would get the freedoms I intended
to give them.
It is not enough to require that the combined program be free software somehow. It
must be released, as a whole, **under the original copyleft license**, to ensure that:
(1) subsequent users get the exact same freedoms and (2) subsequent intermediaries
do not get more opportunity than first-stage intermediaries to make the program nonfree.’
(emphasis mine)
See also <https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/>.
The argument ‘it's the _user_ that's building & linking, not ‘the guix project’’ doesn't
seem convincing to me. We're still linking the kernel and OpenZFS kernel module,
we're just doing it lazily (in the SRFI-45 sense).
Until Sun's copyright expires or Oracle relicenses their ZFS code,
I don't think we can legally provide the ZFS kernel module in guix
(the user space tools should be fine, but they are useless without the kernel
modules I assume, so maybe remove them too?).
> +As a large and complex kernel module, OpenZFS has to be compiled for a
> +specific version of Linux-Libre. At times, the latest OpenZFS package
> +available in Guix is not compatible with the latest Linux-Libre version.
> +Thus, directly installing the @code{zfs} package can fail.
> +
> +Instead, you are recommended to select a specific older long-term-support
> +Linux-Libre kernel. Do not use @code{linux-libre-lts}, as even the
> +latest long-term-support kernel may be too new for @code{zfs}. Instead,
> +explicitly select a specific older version, such as @code{linux-libre-5.10},
> +and upgrade it manually later as new long-term-support kernels become
> +available that you have confirmed is compatible with the latest available
> +OpenZFS version on Guix.
> +
> +For example, you can modify your system configuration file to a specific
> +Linux-Libre version and add the @code{zfs-service-type} service.
> +
> +@lisp
> +(use-modules (gnu))
> +(use-package-modules
> + #;@dots{}
> + linux)
> +(use-service-modules
> + #;@dots{}
> + file-systems)
> +
> +(define my-kernel linux-libre-5.10)
> +
> +(operating-system
> + (kernel my-kernel)
> + #;@dots{}
> + (services
> + (cons* (service zfs-service-type
> + (zfs-configuration
> + (kernel my-kernel)))
> + #;@dots{}
> + %desktop-services))
> + #;@dots{})
> +@end lisp
> +
> +@defvr {Scheme Variable} zfs-service-type
> +This is the type for a service that adds ZFS support to your operating
> +system. The service is configured using a @code{zfs-configuration}
> +record.
> +
> +Here is an example use:
> +
> +@lisp
> +(service zfs-service-type
> + (zfs-configuration
> + (kernel linux-libre-5.4)))
> +@end lisp
> +@end defvr
> +
> +@deftp {Data Type} zfs-configuration
> +This data type represents the configuration of the ZFS support in Guix
> +System. Its fields are:
> +
> +@table @asis
> +@item @code{kernel}
> +The package of the Linux-Libre kernel to compile OpenZFS for. This field
> +is always required. It @emph{must} be the same kernel you use in your
> +@code{operating-system} form.
> +
> +@item @code{base-zfs} (default: @code{zfs})
> +The OpenZFS package that will be compiled for the given Linux-Libre kernel.
> +
> +@item @code{base-zfs-auto-snapshot} (default: @code{zfs-auto-snapshot})
> +The @code{zfs-auto-snapshot} package to use. It will be modified to
> +specifically use the OpenZFS compiled for your kernel.
> +
> +@item @code{dependencies} (default: @code{'()})
> +A list of @code{<file-system>} or @code{<mapped-device>} records that must
> +be mounted or opened before OpenZFS scans for pools to import. For example,
> +if you have set up LUKS containers as leaf VDEVs in a pool, you have to
> +include their corresponding @code{<mapped-ddevice>} records so that OpenZFS
> +can import the pool correctly at bootup.
> +
> +@item @code{auto-mount?} (default: @code{#t})
> +Whether to mount datasets with the ZFS @code{mountpoint} property automatically
> +at startup. This is the behavior that ZFS users usually expect. You might
> +set this to @code{#f} for an operating system intended as a ``rescue'' system
> +that is intended to help debug problems with the disks rather than actually
> +work in production.
> +
> +@item @code{auto-scrub} (default: @code{'weekly})
> +Specifies how often to scrub all pools. Can be the symbols @code{'weekly} or
> +@code{'monthly}, or a schedule specification understood by
> +@xref{mcron, mcron job specifications,, mcron, GNU@tie{}mcron}, such as
> +@code{"0 3 * * 6"} for ``every 3AM on Saturday''.
> +It can also be @code{#f} to disable auto-scrubbing (@strong{not recommended}).
> +
> +The general guideline is to scrub weekly when using consumer-quality drives, and
> +to scrub monthly when using enterprise-quality drives.
> +
> +@code{'weekly} scrubs are done on Sunday midnight, while @code{monthly} scrubs
> +are done on midnight on the first day of each month.
> +
> +@item @code{auto-snapshot?} (default: @code{#t})
> +Specifies whether to auto-snapshot by default. If @code{#t}, then snapshots
> +are automatically created except for ZFS datasets with the
> +@code{com.sun:auto-snapshot} ZFS vendor property set to @code{false}.
> +
> +If @code{#f}, snapshots will not be automatically created, unless the ZFS
> +dataset has the @code{com.sun:auto-snapshot} ZFS vendor property set to
> +@code{true}.
> +
> +@item @code{auto-snapshot-keep} (default: @code{'()})
> +Specifies an association list of symbol-number pairs, indicating the number
> +of automatically-created snapshots to retain for each frequency type.
> +
> +If not specified via this field, by default there are 4 @code{frequent}, 24
> +@code{hourly}, 31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly} snapshots.
> +
> +For example:
> +
> +@lisp
> +(zfs-configuration
> + (kernel my-kernel)
> + (auto-snapshot-keep
> + '((frequent . 8)
> + (hourly . 12))))
> +@end lisp
> +
> +The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
> +@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
> +defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
> +
> +@end table
> +@end deftp
> +
> +@subsubsection ZFS Auto-Snapshot
> +
> +The ZFS service on Guix System supports auto-snapshots as implemented in the
> +Solaris operating system.
> +
> +@code{frequent} (every 15 minutes), @code{hourly}, @code{daily}, @code{weekly},
> +and @code{monthly} snapshots are created automatically for ZFS datasets that
> +have auto-snapshot enabled. They will be named, for example,
> +@code{zfs-auto-snap_frequent-2021-03-22-1415}. You can continue to use
> +manually-created snapshots as long as they do not conflict with the naming
> +convention used by auto-snapshot. You can also safely manually destroy
> +automatically-created snapshots, for example to free up space.
> +
> +The @code{com.sun:auto-snapshot} ZFS property controls auto-snapshot on a
> +per-dataset level. Sub-datasets will inherit this property from their parent
> +dataset, but can have their own property.
> +
> +You @emph{must} set this property to @code{true} or @code{false} exactly,
> +otherwise it will be treated as if the property is unset.
> +
> +For example:
> +
> +@example
> +# zfs list -o name
> +NAME
> +tank
> +tank/important-data
> +tank/tmp
> +# zfs set com.sun:auto-snapshot=true tank
> +# zfs set com.sun:auto-snapshot=false tank/tmp
> +@end example
> +
> +The above will set @code{tank} and @code{tank/important-data} to be
> +auto-snapshot, while @code{tank/tmp} will not be auto-snapshot.
> +
> +If the @code{com.sun:auto-snapshot} property is not set for a dataset
> +(the default when pools and datasets are created), then whether
> +auto-snapshot is done or not will depend on the @code{auto-snapshot?}
> +field of the @code{zfs-configuration} record.
> +
> +There are also @code{com.sun:auto-snapshot:frequent},
> +@code{com.sun:auto-snapshot:hourly}, @code{com.sun:auto-snapshot:daily},
> +@code{com.sun:auto-snapshot:weekly}, and @code{com.sun:auto-snapshot:monthly}
> +properties that give finer-grained control of whether to auto-snapshot a
> +dataset at a particular schedule.
> +
> +The number of snapshots kept for all datasets can be overridden via the
> +@code{auto-snapshot-keep} field of the @code{zfs-configuration} record.
> +There is currently no support to have different numbers of snapshots to
> +keep for different datasets.
> +
> +@subsubsection ZVOLs
> +
> +ZFS supports ZVOLs, block devices that ZFS exposes to the operating
> +system in the @code{/dev/zvol/} directory. The ZVOL will have the same
> +resilience and self-healing properties as other datasets on your ZFS pool.
> +ZVOLs can also be snapshotted (and will be included in auto-snapshotting
> +if enabled), which snapshots the state of the block device, effectively
> +snapshotting the hosted file system.
> +
> +You can put any file system inside the ZVOL. However, in order to mount this
> +file system at system start, you need to add @code{%zfs-zvol-dependency} as a
> +dependency of each file system inside a ZVOL.
> +
> +@defvr {Scheme Variable} %zfs-zvol-dependency
> +An artificial @code{<mapped-device>} which tells the file system mounting
> +service to wait for ZFS to provide ZVOLs before mounting the
> +@code{<file-system>} dependent on it.
> +@end defvr
> +
> +For example, suppose you create a ZVOL and put an ext4 filesystem
> +inside it:
> +
> +@example
> +# zfs create -V 100G tank/ext4-on-zfs
> +# mkfs.ext4 /dev/zvol/tank/ext4-on-zfs
> +# mkdir /ext4-on-zfs
> +# mount /dev/zvol/tank/ext4-on-zfs /ext4-on-zfs
> +@end example
> +
> +You can then set this up to be mounted at boot by adding this to the
> +@code{file-systems} field of your @code{operating-system} record:
> +
> +@lisp
> +(file-system
> + (device "/dev/zvol/tank/ext4-on-zfs")
> + (mount-point "/ext4-on-zfs")
> + (type "ext4")
> + (dependencies (list %zfs-zvol-dependency)))
> +@end lisp
> +
> +You @emph{must not} add @code{%zfs-zvol-dependency} to your
> +@code{operating-system}'s @code{mapped-devices} field, and you @emph{must
> +not} add it (or any @code{<file-system>}s dependent on it) to the
> +@code{dependencies} field of @code{zfs-configuration}. Finally, you
> +@emph{must not} use @code{%zfs-zvol-dependency} unless you actually
> +instantiate @code{zfs-service-type} on your system.
> +
> +@subsubsection Unsupported Features
> +
> +Some common features and uses of ZFS are currently not supported, or not
> +fully supported, on Guix.
> +
> +@enumerate
> +@item
> +Shepherd-managed daemons that are configured to read from or write to ZFS
> +mountpoints need to include @code{user-processes} in their @code{requirement}
> +field. This is the earliest that ZFS file systems are assured of being
> +mounted.
> +
> +Generally, most daemons will, directly or indirectly, require
> +@code{networking}, or @code{user-processes}, or both. Most implementations
> +of @code{networking} will also require @code{user-processes} so daemons that
> +require only @code{networking} will also generally start up after
> +@code{user-processes}. A notable exception, however, is
> +@code{static-networking-service-type}. You will need to explicitly add
> +@code{user-processes} as a @code{requirement} of your @code{static-networking}
> +record.
> +
> +@item
> +@code{mountpoint=legacy} ZFS file systems. The handlers for the Guix mounting
> +system have not yet been modified to support ZFS, and will expect @code{/dev}
> +paths in the @code{<file-system>}'s @code{device} field, but ZFS file systems
> +are referred to via non-path @code{pool/file/system} names. Such file systems
> +also need to be mounted @emph{after} OpenZFS has scanned for pools.
> +
> +You can still manually mount these file systems after system boot; what is
> +only unsupported is mounting them automatically at system boot by specifying
> +them in @code{<file-system>} records of your @code{operating-system}.
> +
> +@item
> +@code{/home} on ZFS. Guix will create home directories for users, but this
> +process currently cannot be scheduled after ZFS file systems are mounted.
> +Thus, the ZFS file system might be mounted @emph{after} Guix has created
> +home directories at boot, at which point OpenZFS will refuse to mount since
> +the mountpoint is not empty. However, you @emph{can} create an ext4, xfs,
> +btrfs, or other supported file system inside a ZVOL, have that depend on
> +@code{%zfs-zvol-dependency}, and set it to mount on the @code{/home}
> +directory; they will be scheduled to mount before the @code{user-homes}
> +process.
> +
> +Similarly, other locations like @code{/var}, @code{/gnu/store} and so
> +on cannot be reliably put in a ZFS file system, though they may be
> +possible to create as other file systems inside ZVOL containers.
> +
> +@item
> +@code{/} and @code{/boot} on ZFS. These require Guix to expose more of
> +the @code{initrd} very early boot process to services. It also requires
> +Guix to have the ability to explicitly load modules while still in
> +@code{initrd} (currently kernel modules loaded by
> +@code{kernel-module-loader-service-type} are loaded after @code{/} is
> +mounted). Further, since one of ZFS's main advantages is that it can
> +continue working despite the loss of one or more devices, it makes sense
> +to also support installing the bootloader on all devices of the pool that
> +contains the @code{/} and @code{/boot}; after all, if ZFS can survive the
> +loss of one device, the bootloader should also be able to survive the loss
> +of one device.
> +
> +@item
> +ZVOL swap devices. Mapped swap devices need to be listed in
> +@code{mapped-devices} to ensure they are opened before the system attempts
> +to use them, but you cannot currently add @code{%zfs-zvol-dependency} to
> +@code{mapped-devices}.
> +
> +This will also require significant amounts of testing, as various kernel
> +build options and patches may affect how swapping works, which are possibly
> +different on Guix System compared to other distributions that this feature is
> +known to work on.
> +
> +@item
> +ZFS Event Daemon. Support for this has not been written yet, patches are
> +welcome. The main issue is how to design this in a Guix style while
> +supporting legacy shell-script styles as well. In particular, OpenZFS itself
> +comes with a number of shell scripts intended for ZFS Event Daemon, and we
> +need to figure out how the user can choose to use or not use the provided
> +scripts (and configure any settings they have) or override with their own
> +custom code (which could be shell scripts they have written and trusted from
> +previous ZFS installations).
> +
> +As-is, you can create your own service that activates the ZFS Event Daemon
> +by creating the @file{/etc/zfs/zed} directory and filling it appropriately,
> +then launching @code{zed}.
> +
> +@item
> +@file{/etc/zfs/zpool.cache}. Currently the ZFS support on Guix always forces
> +scanning of all devices at bootup to look for ZFS pools. For systems with
> +dozens or hundreds of storage devices, this can lead to slow bootup. One issue
> +is that tools should really not write to @code{/etc} which is supposed to be for
> +configuration; possibly it could be moved to @code{/var} instead. Another issue
> +is that if Guix ever supports @code{/} on ZFS, we would need to somehow keep the
> +@code{zpool.cache} file inside the @code{initrd} up-to-date with what is in the
> +@code{/} mount point.
> +
> +@item
> +@code{zfs share}. This will require some (unknown amount of) work to integrate
> +into the Samba and NFS services of Guix. You @emph{can} manually set up Samba
> +and NFS to share any mounted ZFS datasets by setting up their configurations
> +properly; it just can't be done for you by @code{zfs share} and the
> +@code{sharesmb} and @code{sharenfs} properties.
> +@end enumerate
> +
> +Hopefully, support for the above only requires code to be written, o users
o users -> and users
> +are encouraged to hack on Guix to implement the above features.
> +
> @node Mapped Devices
> @section Mapped Devices
I'm not familiar enough with ZFS to understand all that text above; I'll assume
it's fine.
> diff --git a/gnu/local.mk b/gnu/local.mk
> index b944c671af..a2ff871277 100644
> --- a/gnu/local.mk
> +++ b/gnu/local.mk
> @@ -43,6 +43,7 @@
> # Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
> # Copyright © 2021 Arun Isaac <arunisaac@systemreboot.net>
> # Copyright © 2021 Sharlatan Hellseher <sharlatanus@gmail.com>
> +# Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
> #
> # This file is part of GNU Guix.
> #
> @@ -618,6 +619,7 @@ GNU_SYSTEM_MODULES = \
> %D%/services/docker.scm \
> %D%/services/authentication.scm \
> %D%/services/file-sharing.scm \
> + %D%/services/file-systems.scm \
> %D%/services/games.scm \
> %D%/services/ganeti.scm \
> %D%/services/getmail.scm \
> diff --git a/gnu/services/base.scm b/gnu/services/base.scm
> index ab3e441a7b..bcca24f93a 100644
> --- a/gnu/services/base.scm
> +++ b/gnu/services/base.scm
> @@ -185,7 +185,9 @@
>
> references-file
>
> - %base-services))
> + %base-services
> +
> + dependency->shepherd-service-name))
>
> ;;; Commentary:
> ;;;
> diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
> new file mode 100644
> index 0000000000..0b1aae38ac
> --- /dev/null
> +++ b/gnu/services/file-systems.scm
> @@ -0,0 +1,295 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>;.
> +
> +(define-module (gnu services file-systems)
> + #:use-module (gnu packages file-systems)
> + #:use-module (gnu services)
> + #:use-module (gnu services base)
> + #:use-module (gnu services linux)
> + #:use-module (gnu services mcron)
> + #:use-module (gnu services shepherd)
> + #:use-module (gnu system mapped-devices)
> + #:use-module (guix gexp)
> + #:use-module (guix packages)
> + #:use-module (guix records)
> + #:export (zfs-service-type
> +
> + zfs-configuration
> + zfs-configuration?
> + zfs-configuration-kernel
> + zfs-configuration-base-zfs
> + zfs-configuration-base-zfs-auto-snapshot
> + zfs-configuration-dependencies
> + zfs-configuration-auto-mount?
> + zfs-configuration-auto-scrub
> + zfs-configuration-auto-snapshot?
> + zfs-configuration-auto-snapshot-keep
> +
> + %zfs-zvol-dependency))
> +
> +(define-record-type* <zfs-configuration>
> + zfs-configuration
> + make-zfs-configuration
> + zfs-configuration?
> +
> + ; linux-libre kernel you want to compile the base-zfs module for.
If a comment is on a line of its own, it's normally ";;".
";" is usually only for comments on the same line as some expression.
> + (kernel zfs-configuration-kernel)
> + ; the OpenZFS package that will be modified to compile for the
> + ; given kernel.
> + (base-zfs zfs-configuration-base-zfs
> + (default zfs))
> +
> + ; the zfs-auto-snapshot package that will be modified to compile
> + ; for the given kernel.
> + (base-zfs-auto-snapshot zfs-configuration-base-zfs-auto-snapshot
> + (default zfs-auto-snapshot))
> +
> + ; list of <mapped-device> or <file-system> objects that must be
> + ; opened/mounted before we import any ZFS pools.
> + (dependencies zfs-configuration-dependencies
> + (default '()))
> +
> + ; #t if mountable datasets are to be mounted automatically.
Gramatically simpler: #t to mount all mountable datasets by default.
> + ; #f if not mounting.
> + ; #t is the expected behavior on other operating systems, the
> + ; #f is only supported for "rescue" operating systems where
> + ; the user wants lower-level control of when to mount.
> + (auto-mount? zfs-configuration-auto-mount?
> + (default #t))
> +
> + ; 'weekly for weekly scrubbing, 'monthly for monthly scrubbing, an
> + ; mcron time specification that can be given to `job`, or #f to
> + ; disable.
> + (auto-scrub zfs-configuration-auto-scrub
> + (default 'weekly))
> +
> + ; #t if auto-snapshot is default (and `com.sun:auto-snapshot=false`
--> #t if auto-snapshot is the default
or
--> #t to auto-snapshot by default
> + ; disables auto-snapshot per dataset), #f if no auto-snapshotting
> + ; is default (and `com.sun:auto-snapshot=true` enables auto-snapshot
> + ; per dataset).
> + (auto-snapshot? zfs-configuration-auto-snapshot?
> + (default #t))
> +
> + ; association list of symbol-number pairs to indicate the number
> + ; of automatic snapshots to keep for each of 'frequent, 'hourly,
> + ; 'daily, 'weekly, and 'monthly.
> + ; e.g. '((frequent . 8) (hourly . 12))
> + (auto-snapshot-keep zfs-configuration-auto-snapshot-keep
> + (default '())))
Seems a reasonable structure, though I'm not familiar with ZFS.
> +(define %default-auto-snapshot-keep
> + '((frequent . 4)
> + (hourly . 24)
> + (daily . 31)
> + (weekly . 8)
> + (monthly . 12)))
> +
> +(define %auto-snapshot-mcron-schedule
> + '((frequent . "0,15,30,45 * * * *")
> + (hourly . "0 * * * *")
> + (daily . "0 0 * * *")
> + (weekly . "0 0 * * 7")
> + (monthly . "0 0 1 * *")))
> +
> +;; A synthetic and unusable MAPPED-DEVICE intended for use when
> +;; the user has created a mountable filesystem inside a ZFS
> +;; zvol and wants it mounted inside the configuration.scm.
> +(define %zfs-zvol-dependency
> + (mapped-device
> + (source '())
> + (targets '("zvol/*"))
> + (type #f)))
> +
> +(define (make-zfs-package conf)
> + (let ((kernel (zfs-configuration-kernel conf))
> + (base-zfs (zfs-configuration-base-zfs conf)))
> + (package
> + (inherit base-zfs)
> + (arguments (cons* #:linux kernel
> + (package-arguments base-zfs))))))
> +
> +(define (make-zfs-auto-snapshot-package conf)
> + (let ((zfs (make-zfs-package conf))
> + (base-zfs-auto-snapshot (zfs-configuration-base-zfs-auto-snapshot conf)))
> + (package
> + (inherit base-zfs-auto-snapshot)
> + (inputs `(("zfs" ,zfs))))))
> +
> +(define (zfs-loadable-modules conf)
> + (list (list (make-zfs-package conf) "module")))
> +
> +(define (zfs-shepherd-services conf)
> + (let* ((zfs-package (make-zfs-package conf))
> + (zpool (file-append zfs-package "/sbin/zpool"))
> + (zfs (file-append zfs-package "/sbin/zfs"))
> + (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
> + (scheme-modules `((srfi srfi-1)
> + (srfi srfi-34)
> + (srfi srfi-35)
> + (rnrs io ports)
> + ,@%default-modules)))
> + (define zfs-scan
> + (shepherd-service
> + (provision '(zfs-scan))
> + (requirement `(root-file-system
> + kernel-module-loader
> + udev
> + ,@(map dependency->shepherd-service-name
> + (zfs-configuration-dependencies conf))))
> + (documentation "Scans for and imports ZFS pools.")
> + (modules scheme-modules)
> + (start #~(lambda _
> + (guard (c ((message-condition? c)
> + (format (current-error-port)
> + "zfs: error importing pools: ~s~%"
> + (condition-message c))
> + #f))
> + ; TODO: optionally use a cachefile.
> + (invoke #$zpool "import" "-a" "-N"))))
> + ;; Why not one-shot? Because we don't really want to rescan
> + ;; this each time a requiring process is restarted, as scanning
> + ;; can take a long time and a lot of I/O.
> + (stop #~(const #f))))
> +
> + (define device-mapping-zvol/*
> + (shepherd-service
> + (provision '(device-mapping-zvol/*))
> + (requirement '(zfs-scan))
> + (documentation "Waits for all ZFS ZVOLs to be opened.")
> + (modules scheme-modules)
> + (start #~(lambda _
> + (guard (c ((message-condition? c)
> + (format (current-error-port)
> + "zfs: error opening zvols: ~s~%"
> + (condition-message c))
> + #f))
> + (invoke #$zvol_wait))))
> + (stop #~(const #f))))
> +
> + (define zfs-auto-mount
> + (shepherd-service
> + (provision '(zfs-auto-mount))
> + (requirement '(zfs-scan))
> + (documentation "Mounts all non-legacy mounted ZFS filesystems.")
> + (modules scheme-modules)
> + (start #~(lambda _
> + (guard (c ((message-condition? c)
> + (format (current-error-port)
> + "zfs: error mounting file systems: ~s~%"
> + (condition-message c))
> + #f))
> + ;; Output to current-error-port, otherwise the
> + ;; user will not see any prompts for passwords
> + ;; of encrypted datasets.
> + ;; XXX Maybe better to explicitly open /dev/console ?
> + (with-output-to-port (current-error-port)
> + (lambda ()
> + (invoke #$zfs "mount" "-a" "-l"))))))
> + (stop #~(lambda _
> + ;; Make sure that Shepherd does not have a CWD that
> + ;; is a mounted ZFS filesystem, which would prevent
> + ;; unmounting.
> + (chdir "/")
> + (invoke #$zfs "unmount" "-a" "-f")))))
> +
> + `(,zfs-scan
> + ,device-mapping-zvol/*
> + ,@(if (zfs-configuration-auto-mount? conf)
> + `(,zfs-auto-mount)
> + '()))))
Service definition seems reasonable, but agan, I'm no ZFS expert.
> +(define (zfs-user-processes conf)
> + (if (zfs-configuration-auto-mount? conf)
> + '(zfs-auto-mount)
> + '(zfs-scan)))
> +
> +(define (zfs-mcron-auto-snapshot-jobs conf)
> + (let* ((user-auto-snapshot-keep (zfs-configuration-auto-snapshot-keep conf))
> + ;; assoc-ref has earlier entries overriding later ones.
> + (auto-snapshot-keep (append user-auto-snapshot-keep
> + %default-auto-snapshot-keep))
> + (auto-snapshot? (zfs-configuration-auto-snapshot? conf))
> + (zfs-auto-snapshot-package (make-zfs-auto-snapshot-package conf))
> + (zfs-auto-snapshot (file-append zfs-auto-snapshot-package
> + "/sbin/zfs-auto-snapshot")))
> + (map
> + (lambda (label)
> + (let ((keep (assoc-ref auto-snapshot-keep label))
> + (sched (assoc-ref %auto-snapshot-mcron-schedule label)))
> + #~(job '#$sched
> + (string-append #$zfs-auto-snapshot
> + " --quiet --syslog "
> + " --label=" #$(symbol->string label)
> + " --keep=" #$(number->string keep)
> + " //"))))
> + '(frequent hourly daily weekly monthly))))
> +
> +(define (zfs-mcron-auto-scrub-jobs conf)
> + (let* ((zfs-package (make-zfs-package conf))
> + (zpool (file-append zfs-package "/sbin/zpool"))
> + (auto-scrub (zfs-configuration-auto-scrub conf))
> + (sched (cond
> + ((eq? auto-scrub 'weekly) "0 0 * * 7")
> + ((eq? auto-scrub 'monthly) "0 0 1 * *")
> + (else auto-scrub))))
> + (list
> + #~(job '#$sched
> +
> + ;; Suppress errors: if there are no ZFS pools, the
> + ;; scrub will not be given any arguments, which makes
> + ;; it error out.
> + (string-append "(" #$zpool " scrub `" #$zpool " list -o name -H` "
> + "> /dev/null 2>&1) "
> + "|| exit 0")))))
I think people here would prefer something in Guile, e.g. using 'invoke'.
For redirecting to "/dev/null", maybe with-output-to-port, with-error-to-port
and (%make-void-port "w0") can be used. Doing this in Guile would avoid any
potential issues with whitespace and the escape rules for bash.
> +(define (zfs-mcron-jobs conf)
> + (append (zfs-mcron-auto-snapshot-jobs conf)
> + (if (zfs-configuration-auto-scrub conf)
> + (zfs-mcron-auto-scrub-jobs conf)
> + '())))
> +
> +(define zfs-service-type
> + (service-type
> + (name 'zfs)
> + (extensions
> + (list ;; Install OpenZFS kernel module into kernel profile.
> + (service-extension linux-loadable-module-service-type
> + zfs-loadable-modules)
> + ;; And load it.
> + (service-extension kernel-module-loader-service-type
> + (const '("zfs")))
> + ;; Make sure ZFS pools and datasets are mounted at
> + ;; boot.
> + (service-extension shepherd-root-service-type
> + zfs-shepherd-services)
> + ;; Make sure user-processes don't start until
> + ;; after ZFS does.
> + (service-extension user-processes-service-type
> + zfs-user-processes)
> + ;; Install automated scrubbing and snapshotting.
> + (service-extension mcron-service-type
> + zfs-mcron-jobs)
> +
> + ;; Install ZFS management commands in the system
> + ;; profile.
> + (service-extension profile-service-type
> + (compose list make-zfs-package))
FWIW, I (YMMV) expect that the only packages guix installs in the system
profile are those listed in the 'packages' field whenever feasible.
Though adding zfs to the system profile when ZFS file systems seems
reasonable.
Apparently the consensus seems to be that putting the file system utilities
into the system profile automatically is useful and not problematic:
<https://issues.guix.gnu.org/issue/39505>, <https://issues.guix.gnu.org/49128>,
so putting it there is probably ok?
> + ;; Install ZFS udev rules.
> + (service-extension udev-service-type
> + (compose list make-zfs-package))))
> + (description "Installs ZFS, an advanced filesystem and volume manager.")))
Also, a more general comment: ideally most top-level procedures have a docstring
or comment explaining what they are for and how they are used.
Overall, the code seems to be quite reasonable (I haven't tested it though)
with a minor issues.
If it weren't for the GPL violation, I was going to recommend that the patch
would be merged (after resolving the minor issues) --- there is the issue that
there are very few people familiar enough with ZFS to review this, so I thought
it might be reasonable to consider you ‘the guix expert on ZFS’, with the idea
that if someone has ZFS problems, they would be directed to you, and if someone
has ZFS-related patches, you would try to review them, and that over time, guix
will attract some reviewers understanding ZFS well-enough to review ZFS things,
so after some time, you wouldn't have do all that on your own.
The GPL violation is very unfortunate. It would have been nice to have some
ZFS support in Guix.
Greetings,
Maxime
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] Gaslighting
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (8 preceding siblings ...)
2021-07-25 14:31 ` [bug#45692] [PATCH v4 " raid5atemyhomework via Guix-patches via
@ 2021-09-02 21:24 ` Mason Loring Bliss
2021-09-03 12:22 ` Maxime Devos
2021-09-06 7:59 ` [bug#45692] zimoun
2021-09-30 14:56 ` [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type raid5atemyhomework via Guix-patches via
` (2 subsequent siblings)
12 siblings, 2 replies; 81+ messages in thread
From: Mason Loring Bliss @ 2021-09-02 21:24 UTC (permalink / raw)
To: 45692
[-- Attachment #1: Type: text/plain, Size: 950 bytes --]
Maxime writes,
> The GPL violation is very unfortunate. It would have been nice to
> have some ZFS support in Guix.
There is no GPL violation. Gaslighting in an attempt to sabotage the
adoption of high-quality free software is somewhat poor form and not at all
useful.
To be clear, the one singular thing that would be a GPL violation would be
Guix building Linux with ZFS built in and then distributing that binary.
Users can build ZFS into their kernels and use them and that's fine as long
as they don't distribute them. Guix can script this to make it easy for
users to build ZFS into their kernels and that would not be a violation.
The only possible violation is distribution of a binary Linux kernel with
ZFS compiled into it.
Hope that helps.
--
Mason Loring Bliss mason@blisses.org Ewige Blumenkraft!
(if awake 'sleep (aref #(sleep dream) (random 2))) -- Hamlet, Act III, Scene I
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-02 20:57 ` Maxime Devos
@ 2021-09-02 22:22 ` Maxime Devos
2021-09-03 12:41 ` raid5atemyhomework via Guix-patches via
2021-09-04 18:58 ` raid5atemyhomework via Guix-patches via
2021-09-06 8:08 ` zimoun
2 siblings, 1 reply; 81+ messages in thread
From: Maxime Devos @ 2021-09-02 22:22 UTC (permalink / raw)
To: raid5atemyhomework, 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 318 bytes --]
Maxime Devos schreef op do 02-09-2021 om 22:57 [+0200]:
> Hi,
>
> Some comments on the code. Spoiler: the code is presumably good,
> but there's a GPL violation.
Nevermind, it's apparently less of an issue than I expected?
See the links to IRC at <https://issues.guix.gnu.org/50347>.
Greetings,
Maxime.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] Gaslighting
2021-09-02 21:24 ` [bug#45692] Gaslighting Mason Loring Bliss
@ 2021-09-03 12:22 ` Maxime Devos
2021-09-06 7:59 ` [bug#45692] zimoun
1 sibling, 0 replies; 81+ messages in thread
From: Maxime Devos @ 2021-09-03 12:22 UTC (permalink / raw)
To: Mason Loring Bliss, 45692
[-- Attachment #1: Type: text/plain, Size: 2119 bytes --]
Mason Loring Bliss schreef op do 02-09-2021 om 17:24 [-0400]:
> Maxime writes,
>
> > The GPL violation is very unfortunate. It would have been nice to
> > have some ZFS support in Guix.
>
> There is no GPL violation. Gaslighting in an attempt to sabotage the
> adoption of high-quality free software is somewhat poor form and not at all
> useful.
Indeed, gaslighing is poor form and not at all useful. But why are you suggesting
I'm gaslightling here? Did you just read these last two sentences of the e-mail?
If you read all of it, you'd have seen my explanation of why I believe there's
a GPL violation, and two relevant links to articles by SFLC and FSF.
Also see IRC logs:
https://logs.guix.gnu.org/guix/2021-09-02.log
https://logs.guix.gnu.org/guix/2021-09-03.log
While ultimately it's a matter for the courts to decide on,
I believe I've reasonable grounds to believe it's a GPL violation
and explained why, so I don't see any gaslightling here.
And I'm not ‘sabotaging’ anything. In fact, I'm _helping_ adoption of ZFS,
by reviewing the patch and giving some suggestions. There is just the practical
problem of the in-my-eyes probable GPL violation (your opinion on whether it's
a GPL violation might vary).
> To be clear, the one singular thing that would be a GPL violation would be
> Guix building Linux with ZFS built in and then distributing that binary.
> Users can build ZFS into their kernels and use them and that's fine as long
> as they don't distribute them. Guix can script this to make it easy for
> users to build ZFS into their kernels and that would not be a violation.
> The only possible violation is distribution of a binary Linux kernel with
> ZFS compiled into it.
See the previous mail and the IRC logs for why I find this reasoning rather
flimsy.
> Hope that helps.
No, your libel about ‘Maxime is gaslighting’ doesn't help. And your paragraph
about why you think there's no GPL violation here is nothing I haven't read before.
I probably won't be reading and replying to your mails anymore.
Bye,
Maxime
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-02 22:22 ` Maxime Devos
@ 2021-09-03 12:41 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-09-03 12:41 UTC (permalink / raw)
To: Maxime Devos, Efraim Flashner; +Cc: 45692@debbugs.gnu.org
Greetings Maxime,
> Maxime Devos schreef op do 02-09-2021 om 22:57 [+0200]:
>
> > Hi,
> > Some comments on the code. Spoiler: the code is presumably good,
> > but there's a GPL violation.
>
> Nevermind, it's apparently less of an issue than I expected?
> See the links to IRC at https://issues.guix.gnu.org/50347.
Note that this patch does ***not*** add ZFS to the Guix project.
Instead, this patch creates a convenient service that uses the existing `zfs` package and builds the user system so that the user downloads the ZFS source code, compiles it, and links it to the kernel on the system.
In #50347, you refer to this analysis: https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/
I quote this part:
> ## Is The Analysis Different With Source-Only Distribution?
>
> ...
>
> Pure distribution of source with no binaries is undeniably different. When
> distributing source code and no binaries, requirements in those sections of
> GPLv2 and CDDLv1 that cover modification and/or binary (or “Executable”, as
> CDDLv1 calls it) distribution do not activate. Therefore, the analysis is
> simpler, and we find no specific clause in either license that prohibits
> source-only redistribution of Linux and ZFS, even on the same distribution
> media.
This is in line with the analysis already quoted in the documentation added: https://www.fsf.org/licensing/zfs-and-linux
Guix does ***not*** distribute any binaries; see the file `gnu/packages/file-systems.scm` in the **current** `master` branch of Guix:
> `(;; The ZFS kernel module should not be downloaded since the license
> ;; terms don't allow for distributing it, only building it locally.
> #:substitutable? #f
Note that the above code ***predates*** this patch: fe338d7f009 (Efraim Flashner 2019-12-19 11:47:49 +0200 1188)
Also CCing Efraim here --- presumably he had some choice arguments about how `zfs` got into `gnu/packages/file-systems.scm` in the first place.
If the CDDL-GPL incompatibility is problematic, then why is it only being brought up now, why did it not get brought up in 2019, when Efraim was submitting the patch that put the ZFS package into Guix in the first place?
The code in this patch does not do anything that the user cannot do with their own scripts (indeed, I prototyped much of the code in my own `configuration.scm`).
The code in this patch also does not link, directly or indirectly, into the ZFS kernel module.
At worst, the code in this patch executes the binaries that are the output of compilation, but since it is invoked as a separate binary running in a separate process, there is no legal basis for considering this as "linking", as opposed to merely invoking a separate program (if merely invoking a separate program was enough to "link", then Windows cannot run any free software).
Your referred document then makes some speculation that even source distribution might be problematic.
However, Guix does *not* even distribute sources, by my understanding --- the Guix build downloads from https://github.com/openzfs/zfs/releases/ without going through any Guix servers, so this should be even less of a problem.
If anyone is at legal risk, it is github, not Guix, for distributing the sources --- and it would be very strange for Oracle to not go after github for distributing source code that is intended to be linked to GPL code, but go after Guix; Guix is a much more niche project than the openzfs/zfs project on github.
(my understanding is that Oracle implicitly allows the existence of openzfs/zfs, even has some maintainers of the project on their payroll, so it would be very strange to go after software that just downloads the source code from that project, compiles it, links it, and does ***not*** distribute it (`#:substitutable? #f`))
My understanding (and my argument) is that the already-existing code introduced in fe338d7f009 does not represent a distribution of any ZFS code, not even source:
* The existing fe338d7f009 code tells Guix to download from a github server, not from Guix.
* The existing fe338d7f009 code specifically tells Cuirass to not provide a binary substitute, so Guix end-users must download from github, not from any Guix servers.
* Thus, even if source distribution is legally problematic, Guix does not even distribute the source; the existing fe338d7f009 code just downloads it from an existing distributor.
* That Oracle tolerates the continued existence of https://github.com/openzfs/zfs undermines any legal argument Oracle might make if somebody else builds a script that downloads ZFS from https://github.com/openzfs/zfd and compiles it and links it to GPL code locally without redistributing; that project already contains human-readable instructions on how to download, build, and link ZFS to a Linux kernel, the existing fe338d7f009 code merely translated that text to a machine-readable program.
If Oracle thinks this is legally problematic, they should have demanded shutdown of the https://github.com/openzfs/zfs project first, a much more popular project than Guix.
* The actual code in this patch does not directly invoke the ZFS kernel module.
* The actual code in this patch does link the ZFS kernel module to the kernel of the *local* system, but does not make this linked version available to others; it is only available locally on the system of the user that invokes this actual code.
Again, this is merely a translation (to machine-readable text instructions) of human-readable text instructions on how to link the ZFS kernel module to the Linux kernel, text available publicly on https://github.com/openzfs/zfs.
* The actual code in this patch invokes the ZFS tools as separate programs, thus does not link with them.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-02 20:57 ` Maxime Devos
2021-09-02 22:22 ` Maxime Devos
@ 2021-09-04 18:58 ` raid5atemyhomework via Guix-patches via
2021-09-06 8:08 ` zimoun
2 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-09-04 18:58 UTC (permalink / raw)
To: Maxime Devos; +Cc: 45692@debbugs.gnu.org
Hello again Maxime,
> > +OpenZFS currently only supports Linux-Libre and is not available on the
> > +Hurd.
> > +
> > +OpenZFS is free software; unfortunately its license is incompatible with
> > +the GNU General Public License (GPL), the license of the Linux kernel,
> > +which means they cannot be distributed together. However, as a user,
> > +you can choose to build ZFS and use it together with Linux; you can
> > +even rely on Guix to automate this task. See
> > +@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
> > +the Free Software Foundation} for more information.
>
> That analysis says explicitely that the CDDL is incompatible with the GPL,
> and that they cannot be legally linked together. E.g., see the second quoted
> paragraph:
>
> ‘A copyleft license, including any version of the GNU GPL or GNU AGPL, requires
> augmented versions to be free under the same license -- the same requirement it
> applies to modification of the code.1 I wrote a copyleft license for GNU programs
> to ensure that all users of all versions of them would get the freedoms I intended
> to give them.
>
> It is not enough to require that the combined program be free software somehow. It
> must be released, as a whole, under the original copyleft license, to ensure that:
> (1) subsequent users get the exact same freedoms and (2) subsequent intermediaries
> do not get more opportunity than first-stage intermediaries to make the program nonfree.’
I think the key word you miss here is "released", i.e. "It must be released, as a whole, under the original copyleft license."
Looking at the GPLv2, the word "release" is never used, however I believe the "release" word in the FSF analysis would be considered as a synonym of "distribute" in this context.
The GPLv2 mentions "distribute" many times, but provides no definition of the word.
My understanding is that "distribute" used in GPL means "to provide or make available to at least one person that asks for a copy from you, via some medium".
The GPLv2 imposes many restrictions on the ability to "distribute", so it seems reasonable to consider it an important point.
Now, as I have pointed out, the existing package definition in `gnu/packages/file-systems.scm` specifically disables making a binary copy ("substitute" in Guix parlance) available.
In addition, my understanding is that when compiling from source, the `source` field is what is used, and the `source` field in the `gnu/packages/file-systems.scm` refers to github.com, not any Guix server.
There is no text in GPLv2 which restricts compilation.
However, it can be argued that compilation is a form of translation from source code to machine-executable binary, and the text does mention "translation is included without limitation in the term 'modification'".
GPLv2 restricts modification with three terms:
a. You should have prominent notices on modified files.
b. Extra restriction on copies you ***distribute*** or ***publish***.
c. Extra restriction if the program is interactive and prints copyright notices normally.
(a) does not apply since the linking process used (a form of dynamic linking) does not actually modify any files; presumably only in-memory tables or some such are modified.
(b) does not apply if "distribute" is not what is being done by Guix here.
(c) does not apply since the Linux kernel is not interactive (and even so, does not print copyright notices, not even in debug logs).
On the CDDL side, neither "compile" nor "translate" is ever used, but for completeness let us consider compile == translate == modify.
Modifications are specifically allowed under conditions in section 3.
However, again, section 3 is titled "distribution obligations", meaning they only apply on *distribution*.
So I think the issue here really is: Does Guix "distribute" the ZFS linked with Linux?
My understanding is that the mere existence of code to perform that linking does not in fact *distribute* the code (linking is not the same as distributing); I believe the key point of "distribute" is that a third party gets a copy. And at least, the code I added in this patch does not provide any copy of the compiled code to anyone else; it just stores it on the local machine's disk, in a cpio archive that is used in the system's bootup. The copy, in the execution of the code I added, is never provided to anyone else, so I think my patch is unproblematic
Quick question: does `guix publish` respect `#:substitutable? #f`? If `guix publish` respects `#:substitutable? #f` then it seems to me that even the point "Guix should make at least an attempt to warn users of possibly legal gray areas when distributing" does not apply, too: Guix by itself would not (should not, really; that should be the point of `#:substituable? #f`) publish the compiled code anyway, users who specifically want to publish ZFS and Linux linked together would need to modify the `guix publish` code, and such a user would be running a fork of Guix, not Guix itself, thus should be made aware of this.
Indeed, if `guix publish` does *not* respect `#:substitutable? #f`, I think it would be more effective to protect users against potential legal gray areas for `guix publish` to respect that flag, and for us to then audit existing non-GPLv2-compatible kernel modules and ensure they are `#:substitutable? #f`, then to put up a warning; a warning might be overlooked, but an outright refusal to publish cannot.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-07-25 14:31 ` [bug#45692] [PATCH v4 " raid5atemyhomework via Guix-patches via
2021-08-01 9:41 ` raid5atemyhomework via Guix-patches via
2021-09-02 20:57 ` Maxime Devos
@ 2021-09-04 21:19 ` Xinglu Chen
2021-09-06 10:52 ` raid5atemyhomework via Guix-patches via
2 siblings, 1 reply; 81+ messages in thread
From: Xinglu Chen @ 2021-09-04 21:19 UTC (permalink / raw)
To: raid5atemyhomework, 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 36459 bytes --]
On Sun, Jul 25 2021, raid5atemyhomework via Guix-patches via wrote:
> Hello nonexistent reviewer,
>
> I updated this patch to latest `origin/master` because the previous
> version has bitrotted and will not `git am` cleanly anymore.
>
> There are no changes relative to v3, just rebased it so that the patch applies cleanly.
>
> Testing has been very minimal: I created a VM with the added service,
> then ran it in a VM session that included additional devices (qcow2
> files) from a previous VM run that formatted those as devices of a ZFS
> pool, and confirmed that the new VM could read it and manage that
> pool.
>
>
> Is there any chance this will get reviewed or should I just not bother
> and move on with my life and forget about Guix?
>
>
> At this point as well, I would like to point out what I think is a
> failing of how the Guix distribution is managed.
>
> Guix does not assign any particular committers to any particular tasks or areas.
> The intent is that any committer can review and commit any particular patch.
>
> However, this fails as the Guix project has grown.
>
> No single committer wants to review ***all*** the available patches,
> and this is understandable, as the Guix project has grown
> significantly and includes a wide variety of people with diverging
> interests and as an open-source project you cannot really require that
> particular people look at particular things. Unfortunately, I do not
> know *who* committers are, and more importantly, *which* committer
> might be interested in this ZFS service type. Because "any committer
> can review and commit any patch!!" there is no particular list or
> table I can refer to, to figure out who might be useful to ping for
> this patchset.
>
> At the same time, because no committer is interested in *all* patches
> I cannot just ping some particular person and expect to somehow get on
> some list somewhere that tells me "you will be the 48486th patch that
> will be reviewed by <foo> who is interested in all patches".
>
>
> It is very discouraging to work on this code for a few weeks, release
> it, not get any reviews, and end up in a situation where I have to
> make annoying small changes just to keep the patch from bitrotting.
>
> I understand that there are few possible reviewers, but if potential
> new contributors get discouraged from contributing because they do not
> see their code actually getting in, then you really cannot expect the
> number of reviewers to increase, either.
>
> I think it would be nice if I could at least be told some number of
> people who *might* be interested in this patch, or just throw the
> towel and not bother.
You might want to bring up the topic of subsystem maintainers on the
guix-devel mailing list to get some more attention.
I am just some random ZFS user, so maybe take my comments with a pinch
of salt.
I haven’t followed the thread that closely, so apologies if some of my
questions have already been answered.
>
> Thanks
> raid5atemyhomework
>
>
> From 5351aa7c1c14d4fea032adad895c436e02d1f261 Mon Sep 17 00:00:00 2001
> From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
> Date: Mon, 22 Mar 2021 16:26:28 +0800
> Subject: [PATCH] gnu: Add ZFS service type.
>
> * gnu/services/file-systems.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
> * gnu/services/base.scm: Export dependency->shepherd-service-name.
> * doc/guix.texi (ZFS File System): New subsection.
> ---
> doc/guix.texi | 351 ++++++++++++++++++++++++++++++++++
> gnu/local.mk | 2 +
> gnu/services/base.scm | 4 +-
> gnu/services/file-systems.scm | 295 ++++++++++++++++++++++++++++
> 4 files changed, 651 insertions(+), 1 deletion(-)
> create mode 100644 gnu/services/file-systems.scm
>
> diff --git a/doc/guix.texi b/doc/guix.texi
> index b3c16e6507..e21c47d7ca 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@*
> Copyright @copyright{} 2021 Raghav Gururajan@*
> Copyright @copyright{} 2021 Domagoj Stolfa@*
> Copyright @copyright{} 2021 Hui Lu@*
> +Copyright @copyright{} 2021 raid5atemyhomework@*
>
> Permission is granted to copy, distribute and/or modify this document
> under the terms of the GNU Free Documentation License, Version 1.3 or
> @@ -14265,6 +14266,356 @@ a file system declaration such as:
> compress-force=zstd,space_cache=v2"))
> @end lisp
>
> +
> +@node ZFS File System
> +@subsection ZFS File System
> +
> +Support for ZFS file systems is provided on Guix by the OpenZFS project.
> +OpenZFS currently only supports Linux-Libre and is not available on the
> +Hurd.
> +
> +OpenZFS is free software; unfortunately its license is incompatible with
> +the GNU General Public License (GPL), the license of the Linux kernel,
> +which means they cannot be distributed together. However, as a user,
> +you can choose to build ZFS and use it together with Linux; you can
> +even rely on Guix to automate this task. See
> +@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
> +the Free Software Foundation} for more information.
> +
> +As a large and complex kernel module, OpenZFS has to be compiled for a
> +specific version of Linux-Libre. At times, the latest OpenZFS package
> +available in Guix is not compatible with the latest Linux-Libre version.
> +Thus, directly installing the @code{zfs} package can fail.
> +
> +Instead, you are recommended to select a specific older long-term-support
> +Linux-Libre kernel. Do not use @code{linux-libre-lts}, as even the
> +latest long-term-support kernel may be too new for @code{zfs}. Instead,
> +explicitly select a specific older version, such as @code{linux-libre-5.10},
> +and upgrade it manually later as new long-term-support kernels become
> +available that you have confirmed is compatible with the latest available
> +OpenZFS version on Guix.
> +
> +For example, you can modify your system configuration file to a specific
> +Linux-Libre version and add the @code{zfs-service-type} service.
> +
> +@lisp
> +(use-modules (gnu))
> +(use-package-modules
> + #;@dots{}
> + linux)
> +(use-service-modules
> + #;@dots{}
> + file-systems)
> +
> +(define my-kernel linux-libre-5.10)
> +
> +(operating-system
> + (kernel my-kernel)
> + #;@dots{}
> + (services
> + (cons* (service zfs-service-type
> + (zfs-configuration
> + (kernel my-kernel)))
> + #;@dots{}
> + %desktop-services))
> + #;@dots{})
> +@end lisp
> +
> +@defvr {Scheme Variable} zfs-service-type
> +This is the type for a service that adds ZFS support to your operating
> +system. The service is configured using a @code{zfs-configuration}
> +record.
> +
> +Here is an example use:
> +
> +@lisp
> +(service zfs-service-type
> + (zfs-configuration
> + (kernel linux-libre-5.4)))
> +@end lisp
> +@end defvr
> +
> +@deftp {Data Type} zfs-configuration
> +This data type represents the configuration of the ZFS support in Guix
> +System. Its fields are:
> +
> +@table @asis
> +@item @code{kernel}
> +The package of the Linux-Libre kernel to compile OpenZFS for. This field
> +is always required. It @emph{must} be the same kernel you use in your
> +@code{operating-system} form.
> +
> +@item @code{base-zfs} (default: @code{zfs})
> +The OpenZFS package that will be compiled for the given Linux-Libre kernel.
> +
> +@item @code{base-zfs-auto-snapshot} (default: @code{zfs-auto-snapshot})
> +The @code{zfs-auto-snapshot} package to use. It will be modified to
> +specifically use the OpenZFS compiled for your kernel.
> +
> +@item @code{dependencies} (default: @code{'()})
> +A list of @code{<file-system>} or @code{<mapped-device>} records that must
> +be mounted or opened before OpenZFS scans for pools to import. For example,
> +if you have set up LUKS containers as leaf VDEVs in a pool, you have to
> +include their corresponding @code{<mapped-ddevice>} records so that OpenZFS
> +can import the pool correctly at bootup.
> +
> +@item @code{auto-mount?} (default: @code{#t})
> +Whether to mount datasets with the ZFS @code{mountpoint} property automatically
> +at startup. This is the behavior that ZFS users usually expect. You might
> +set this to @code{#f} for an operating system intended as a ``rescue'' system
> +that is intended to help debug problems with the disks rather than actually
> +work in production.
> +
> +@item @code{auto-scrub} (default: @code{'weekly})
> +Specifies how often to scrub all pools. Can be the symbols @code{'weekly} or
> +@code{'monthly}, or a schedule specification understood by
> +@xref{mcron, mcron job specifications,, mcron, GNU@tie{}mcron}, such as
> +@code{"0 3 * * 6"} for ``every 3AM on Saturday''.
> +It can also be @code{#f} to disable auto-scrubbing (@strong{not recommended}).
> +
> +The general guideline is to scrub weekly when using consumer-quality drives, and
> +to scrub monthly when using enterprise-quality drives.
> +
> +@code{'weekly} scrubs are done on Sunday midnight, while @code{monthly} scrubs
> +are done on midnight on the first day of each month.
> +
> +@item @code{auto-snapshot?} (default: @code{#t})
> +Specifies whether to auto-snapshot by default. If @code{#t}, then snapshots
> +are automatically created except for ZFS datasets with the
> +@code{com.sun:auto-snapshot} ZFS vendor property set to @code{false}.
> +
> +If @code{#f}, snapshots will not be automatically created, unless the ZFS
> +dataset has the @code{com.sun:auto-snapshot} ZFS vendor property set to
> +@code{true}.
> +
> +@item @code{auto-snapshot-keep} (default: @code{'()})
> +Specifies an association list of symbol-number pairs, indicating the number
> +of automatically-created snapshots to retain for each frequency type.
> +
> +If not specified via this field, by default there are 4 @code{frequent}, 24
> +@code{hourly}, 31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly} snapshots.
> +
> +For example:
> +
> +@lisp
> +(zfs-configuration
> + (kernel my-kernel)
> + (auto-snapshot-keep
> + '((frequent . 8)
> + (hourly . 12))))
> +@end lisp
> +
> +The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
> +@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
> +defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
> +
> +@end table
> +@end deftp
IIUC, there is not way specify ZFS pools in the ‘file-systems’ field of
an <operating-system> record. Does this mean that ZFS as the root file
system is not supported, and if so, is there a particular reason for
this?
> +
> +@subsubsection ZFS Auto-Snapshot
> +
> +The ZFS service on Guix System supports auto-snapshots as implemented in the
> +Solaris operating system.
> +
> +@code{frequent} (every 15 minutes), @code{hourly}, @code{daily}, @code{weekly},
> +and @code{monthly} snapshots are created automatically for ZFS datasets that
> +have auto-snapshot enabled. They will be named, for example,
> +@code{zfs-auto-snap_frequent-2021-03-22-1415}. You can continue to use
> +manually-created snapshots as long as they do not conflict with the naming
> +convention used by auto-snapshot. You can also safely manually destroy
> +automatically-created snapshots, for example to free up space.
> +
> +The @code{com.sun:auto-snapshot} ZFS property controls auto-snapshot on a
> +per-dataset level. Sub-datasets will inherit this property from their parent
> +dataset, but can have their own property.
> +
> +You @emph{must} set this property to @code{true} or @code{false} exactly,
> +otherwise it will be treated as if the property is unset.
> +
> +For example:
> +
> +@example
> +# zfs list -o name
> +NAME
> +tank
> +tank/important-data
> +tank/tmp
> +# zfs set com.sun:auto-snapshot=true tank
> +# zfs set com.sun:auto-snapshot=false tank/tmp
> +@end example
> +
> +The above will set @code{tank} and @code{tank/important-data} to be
> +auto-snapshot, while @code{tank/tmp} will not be auto-snapshot.
> +
> +If the @code{com.sun:auto-snapshot} property is not set for a dataset
> +(the default when pools and datasets are created), then whether
> +auto-snapshot is done or not will depend on the @code{auto-snapshot?}
> +field of the @code{zfs-configuration} record.
> +
> +There are also @code{com.sun:auto-snapshot:frequent},
> +@code{com.sun:auto-snapshot:hourly}, @code{com.sun:auto-snapshot:daily},
> +@code{com.sun:auto-snapshot:weekly}, and @code{com.sun:auto-snapshot:monthly}
> +properties that give finer-grained control of whether to auto-snapshot a
> +dataset at a particular schedule.
> +
> +The number of snapshots kept for all datasets can be overridden via the
> +@code{auto-snapshot-keep} field of the @code{zfs-configuration} record.
> +There is currently no support to have different numbers of snapshots to
> +keep for different datasets.
> +
> +@subsubsection ZVOLs
> +
> +ZFS supports ZVOLs, block devices that ZFS exposes to the operating
> +system in the @code{/dev/zvol/} directory. The ZVOL will have the same
> +resilience and self-healing properties as other datasets on your ZFS pool.
> +ZVOLs can also be snapshotted (and will be included in auto-snapshotting
> +if enabled), which snapshots the state of the block device, effectively
> +snapshotting the hosted file system.
> +
> +You can put any file system inside the ZVOL. However, in order to mount this
> +file system at system start, you need to add @code{%zfs-zvol-dependency} as a
> +dependency of each file system inside a ZVOL.
> +
> +@defvr {Scheme Variable} %zfs-zvol-dependency
> +An artificial @code{<mapped-device>} which tells the file system mounting
> +service to wait for ZFS to provide ZVOLs before mounting the
> +@code{<file-system>} dependent on it.
> +@end defvr
> +
> +For example, suppose you create a ZVOL and put an ext4 filesystem
> +inside it:
> +
> +@example
> +# zfs create -V 100G tank/ext4-on-zfs
> +# mkfs.ext4 /dev/zvol/tank/ext4-on-zfs
> +# mkdir /ext4-on-zfs
> +# mount /dev/zvol/tank/ext4-on-zfs /ext4-on-zfs
> +@end example
> +
> +You can then set this up to be mounted at boot by adding this to the
> +@code{file-systems} field of your @code{operating-system} record:
> +
> +@lisp
> +(file-system
> + (device "/dev/zvol/tank/ext4-on-zfs")
> + (mount-point "/ext4-on-zfs")
> + (type "ext4")
> + (dependencies (list %zfs-zvol-dependency)))
> +@end lisp
> +
> +You @emph{must not} add @code{%zfs-zvol-dependency} to your
> +@code{operating-system}'s @code{mapped-devices} field, and you @emph{must
> +not} add it (or any @code{<file-system>}s dependent on it) to the
> +@code{dependencies} field of @code{zfs-configuration}. Finally, you
> +@emph{must not} use @code{%zfs-zvol-dependency} unless you actually
> +instantiate @code{zfs-service-type} on your system.
I am not familiar with ZVOLs, so I can’t really comment on that.
> +@subsubsection Unsupported Features
> +
> +Some common features and uses of ZFS are currently not supported, or not
> +fully supported, on Guix.
> +
> +@enumerate
> +@item
> +Shepherd-managed daemons that are configured to read from or write to ZFS
> +mountpoints need to include @code{user-processes} in their @code{requirement}
> +field. This is the earliest that ZFS file systems are assured of being
> +mounted.
> +
> +Generally, most daemons will, directly or indirectly, require
> +@code{networking}, or @code{user-processes}, or both. Most implementations
> +of @code{networking} will also require @code{user-processes} so daemons that
> +require only @code{networking} will also generally start up after
> +@code{user-processes}. A notable exception, however, is
> +@code{static-networking-service-type}. You will need to explicitly add
> +@code{user-processes} as a @code{requirement} of your @code{static-networking}
> +record.
> +
> +@item
> +@code{mountpoint=legacy} ZFS file systems. The handlers for the Guix mounting
> +system have not yet been modified to support ZFS, and will expect @code{/dev}
> +paths in the @code{<file-system>}'s @code{device} field, but ZFS file systems
> +are referred to via non-path @code{pool/file/system} names. Such file systems
> +also need to be mounted @emph{after} OpenZFS has scanned for pools.
> +
> +You can still manually mount these file systems after system boot; what is
> +only unsupported is mounting them automatically at system boot by specifying
> +them in @code{<file-system>} records of your @code{operating-system}.
> +
> +@item
> +@code{/home} on ZFS. Guix will create home directories for users, but this
> +process currently cannot be scheduled after ZFS file systems are mounted.
> +Thus, the ZFS file system might be mounted @emph{after} Guix has created
> +home directories at boot, at which point OpenZFS will refuse to mount since
> +the mountpoint is not empty. However, you @emph{can} create an ext4, xfs,
> +btrfs, or other supported file system inside a ZVOL, have that depend on
> +@code{%zfs-zvol-dependency}, and set it to mount on the @code{/home}
> +directory; they will be scheduled to mount before the @code{user-homes}
> +process.
> +
> +Similarly, other locations like @code{/var}, @code{/gnu/store} and so
> +on cannot be reliably put in a ZFS file system, though they may be
> +possible to create as other file systems inside ZVOL containers.
> +
> +@item
> +@code{/} and @code{/boot} on ZFS. These require Guix to expose more of
> +the @code{initrd} very early boot process to services. It also requires
> +Guix to have the ability to explicitly load modules while still in
> +@code{initrd} (currently kernel modules loaded by
> +@code{kernel-module-loader-service-type} are loaded after @code{/} is
> +mounted). Further, since one of ZFS's main advantages is that it can
> +continue working despite the loss of one or more devices, it makes sense
> +to also support installing the bootloader on all devices of the pool that
> +contains the @code{/} and @code{/boot}; after all, if ZFS can survive the
> +loss of one device, the bootloader should also be able to survive the loss
> +of one device.
Ah, OK, this answered my previous question.
> +@item
> +ZVOL swap devices. Mapped swap devices need to be listed in
> +@code{mapped-devices} to ensure they are opened before the system attempts
> +to use them, but you cannot currently add @code{%zfs-zvol-dependency} to
> +@code{mapped-devices}.
> +
> +This will also require significant amounts of testing, as various kernel
> +build options and patches may affect how swapping works, which are possibly
> +different on Guix System compared to other distributions that this feature is
> +known to work on.
> +
> +@item
> +ZFS Event Daemon. Support for this has not been written yet, patches are
> +welcome. The main issue is how to design this in a Guix style while
> +supporting legacy shell-script styles as well. In particular, OpenZFS itself
> +comes with a number of shell scripts intended for ZFS Event Daemon, and we
> +need to figure out how the user can choose to use or not use the provided
> +scripts (and configure any settings they have) or override with their own
> +custom code (which could be shell scripts they have written and trusted from
> +previous ZFS installations).
> +
> +As-is, you can create your own service that activates the ZFS Event Daemon
> +by creating the @file{/etc/zfs/zed} directory and filling it appropriately,
> +then launching @code{zed}.
> +
> +@item
> +@file{/etc/zfs/zpool.cache}. Currently the ZFS support on Guix always forces
> +scanning of all devices at bootup to look for ZFS pools. For systems with
> +dozens or hundreds of storage devices, this can lead to slow bootup. One issue
> +is that tools should really not write to @code{/etc} which is supposed to be for
> +configuration; possibly it could be moved to @code{/var} instead. Another issue
> +is that if Guix ever supports @code{/} on ZFS, we would need to somehow keep the
> +@code{zpool.cache} file inside the @code{initrd} up-to-date with what is in the
> +@code{/} mount point.
> +
> +@item
> +@code{zfs share}. This will require some (unknown amount of) work to integrate
> +into the Samba and NFS services of Guix. You @emph{can} manually set up Samba
> +and NFS to share any mounted ZFS datasets by setting up their configurations
> +properly; it just can't be done for you by @code{zfs share} and the
> +@code{sharesmb} and @code{sharenfs} properties.
> +@end enumerate
> +
> +Hopefully, support for the above only requires code to be written, o users
> +are encouraged to hack on Guix to implement the above features.
> +
> @node Mapped Devices
> @section Mapped Devices
>
> diff --git a/gnu/local.mk b/gnu/local.mk
> index b944c671af..a2ff871277 100644
> --- a/gnu/local.mk
> +++ b/gnu/local.mk
> @@ -43,6 +43,7 @@
> # Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
> # Copyright © 2021 Arun Isaac <arunisaac@systemreboot.net>
> # Copyright © 2021 Sharlatan Hellseher <sharlatanus@gmail.com>
> +# Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
> #
> # This file is part of GNU Guix.
> #
> @@ -618,6 +619,7 @@ GNU_SYSTEM_MODULES = \
> %D%/services/docker.scm \
> %D%/services/authentication.scm \
> %D%/services/file-sharing.scm \
> + %D%/services/file-systems.scm \
> %D%/services/games.scm \
> %D%/services/ganeti.scm \
> %D%/services/getmail.scm \
> diff --git a/gnu/services/base.scm b/gnu/services/base.scm
> index ab3e441a7b..bcca24f93a 100644
> --- a/gnu/services/base.scm
> +++ b/gnu/services/base.scm
> @@ -185,7 +185,9 @@
>
> references-file
>
> - %base-services))
> + %base-services
> +
> + dependency->shepherd-service-name))
>
> ;;; Commentary:
> ;;;
> diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
> new file mode 100644
> index 0000000000..0b1aae38ac
> --- /dev/null
> +++ b/gnu/services/file-systems.scm
> @@ -0,0 +1,295 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
> +
> +(define-module (gnu services file-systems)
> + #:use-module (gnu packages file-systems)
> + #:use-module (gnu services)
> + #:use-module (gnu services base)
> + #:use-module (gnu services linux)
> + #:use-module (gnu services mcron)
> + #:use-module (gnu services shepherd)
> + #:use-module (gnu system mapped-devices)
> + #:use-module (guix gexp)
> + #:use-module (guix packages)
> + #:use-module (guix records)
> + #:export (zfs-service-type
> +
> + zfs-configuration
> + zfs-configuration?
> + zfs-configuration-kernel
> + zfs-configuration-base-zfs
> + zfs-configuration-base-zfs-auto-snapshot
> + zfs-configuration-dependencies
> + zfs-configuration-auto-mount?
> + zfs-configuration-auto-scrub
> + zfs-configuration-auto-snapshot?
> + zfs-configuration-auto-snapshot-keep
> +
> + %zfs-zvol-dependency))
> +
> +(define-record-type* <zfs-configuration>
> + zfs-configuration
> + make-zfs-configuration
> + zfs-configuration?
> +
> + ; linux-libre kernel you want to compile the base-zfs module for.
> + (kernel zfs-configuration-kernel)
> +
> + ; the OpenZFS package that will be modified to compile for the
> + ; given kernel.
> + (base-zfs zfs-configuration-base-zfs
> + (default zfs))
The field name usually just contains the package name, so ‘zfs’ and
‘zfs-auto-snapshot’ instead of ‘base-zfs’ and ‘base-zfs-auto-snapshot’.
> + ; the zfs-auto-snapshot package that will be modified to compile
> + ; for the given kernel.
> + (base-zfs-auto-snapshot zfs-configuration-base-zfs-auto-snapshot
> + (default zfs-auto-snapshot))
> +
> + ; list of <mapped-device> or <file-system> objects that must be
> + ; opened/mounted before we import any ZFS pools.
> + (dependencies zfs-configuration-dependencies
> + (default '()))
> +
> + ; #t if mountable datasets are to be mounted automatically.
> + ; #f if not mounting.
> + ; #t is the expected behavior on other operating systems, the
> + ; #f is only supported for "rescue" operating systems where
> + ; the user wants lower-level control of when to mount.
> + (auto-mount? zfs-configuration-auto-mount?
> + (default #t))
> +
> + ; 'weekly for weekly scrubbing, 'monthly for monthly scrubbing, an
> + ; mcron time specification that can be given to `job`, or #f to
> + ; disable.
> + (auto-scrub zfs-configuration-auto-scrub
> + (default 'weekly))
> +
> + ; #t if auto-snapshot is default (and `com.sun:auto-snapshot=false`
> + ; disables auto-snapshot per dataset), #f if no auto-snapshotting
> + ; is default (and `com.sun:auto-snapshot=true` enables auto-snapshot
> + ; per dataset).
> + (auto-snapshot? zfs-configuration-auto-snapshot?
> + (default #t))
> +
> + ; association list of symbol-number pairs to indicate the number
> + ; of automatic snapshots to keep for each of 'frequent, 'hourly,
> + ; 'daily, 'weekly, and 'monthly.
> + ; e.g. '((frequent . 8) (hourly . 12))
> + (auto-snapshot-keep zfs-configuration-auto-snapshot-keep
> + (default '())))
> +
> +(define %default-auto-snapshot-keep
> + '((frequent . 4)
> + (hourly . 24)
> + (daily . 31)
> + (weekly . 8)
> + (monthly . 12)))
> +
> +(define %auto-snapshot-mcron-schedule
> + '((frequent . "0,15,30,45 * * * *")
> + (hourly . "0 * * * *")
> + (daily . "0 0 * * *")
> + (weekly . "0 0 * * 7")
> + (monthly . "0 0 1 * *")))
> +
> +;; A synthetic and unusable MAPPED-DEVICE intended for use when
> +;; the user has created a mountable filesystem inside a ZFS
> +;; zvol and wants it mounted inside the configuration.scm.
> +(define %zfs-zvol-dependency
> + (mapped-device
> + (source '())
> + (targets '("zvol/*"))
> + (type #f)))
> +
> +(define (make-zfs-package conf)
> + (let ((kernel (zfs-configuration-kernel conf))
> + (base-zfs (zfs-configuration-base-zfs conf)))
> + (package
> + (inherit base-zfs)
> + (arguments (cons* #:linux kernel
> + (package-arguments base-zfs))))))
> +
> +(define (make-zfs-auto-snapshot-package conf)
> + (let ((zfs (make-zfs-package conf))
> + (base-zfs-auto-snapshot (zfs-configuration-base-zfs-auto-snapshot conf)))
> + (package
> + (inherit base-zfs-auto-snapshot)
> + (inputs `(("zfs" ,zfs))))))
> +
> +(define (zfs-loadable-modules conf)
> + (list (list (make-zfs-package conf) "module")))
> +
> +(define (zfs-shepherd-services conf)
> + (let* ((zfs-package (make-zfs-package conf))
> + (zpool (file-append zfs-package "/sbin/zpool"))
> + (zfs (file-append zfs-package "/sbin/zfs"))
> + (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
> + (scheme-modules `((srfi srfi-1)
> + (srfi srfi-34)
> + (srfi srfi-35)
> + (rnrs io ports)
> + ,@%default-modules)))
> + (define zfs-scan
> + (shepherd-service
> + (provision '(zfs-scan))
> + (requirement `(root-file-system
> + kernel-module-loader
> + udev
> + ,@(map dependency->shepherd-service-name
> + (zfs-configuration-dependencies conf))))
> + (documentation "Scans for and imports ZFS pools.")
> + (modules scheme-modules)
> + (start #~(lambda _
> + (guard (c ((message-condition? c)
> + (format (current-error-port)
> + "zfs: error importing pools: ~s~%"
> + (condition-message c))
> + #f))
> + ; TODO: optionally use a cachefile.
> + (invoke #$zpool "import" "-a" "-N"))))
> + ;; Why not one-shot? Because we don't really want to rescan
> + ;; this each time a requiring process is restarted, as scanning
> + ;; can take a long time and a lot of I/O.
> + (stop #~(const #f))))
> +
> + (define device-mapping-zvol/*
> + (shepherd-service
> + (provision '(device-mapping-zvol/*))
> + (requirement '(zfs-scan))
> + (documentation "Waits for all ZFS ZVOLs to be opened.")
> + (modules scheme-modules)
> + (start #~(lambda _
> + (guard (c ((message-condition? c)
> + (format (current-error-port)
> + "zfs: error opening zvols: ~s~%"
> + (condition-message c))
> + #f))
> + (invoke #$zvol_wait))))
> + (stop #~(const #f))))
> +
> + (define zfs-auto-mount
> + (shepherd-service
> + (provision '(zfs-auto-mount))
> + (requirement '(zfs-scan))
> + (documentation "Mounts all non-legacy mounted ZFS filesystems.")
> + (modules scheme-modules)
> + (start #~(lambda _
> + (guard (c ((message-condition? c)
> + (format (current-error-port)
> + "zfs: error mounting file systems: ~s~%"
> + (condition-message c))
> + #f))
> + ;; Output to current-error-port, otherwise the
> + ;; user will not see any prompts for passwords
> + ;; of encrypted datasets.
> + ;; XXX Maybe better to explicitly open /dev/console ?
Seeing this comment, I assume that encrypted pools are supported, right?
> + (with-output-to-port (current-error-port)
> + (lambda ()
> + (invoke #$zfs "mount" "-a" "-l"))))))
> + (stop #~(lambda _
> + ;; Make sure that Shepherd does not have a CWD that
> + ;; is a mounted ZFS filesystem, which would prevent
> + ;; unmounting.
> + (chdir "/")
> + (invoke #$zfs "unmount" "-a" "-f")))))
> +
> + `(,zfs-scan
> + ,device-mapping-zvol/*
> + ,@(if (zfs-configuration-auto-mount? conf)
> + `(,zfs-auto-mount)
> + '()))))
> +
> +(define (zfs-user-processes conf)
> + (if (zfs-configuration-auto-mount? conf)
> + '(zfs-auto-mount)
> + '(zfs-scan)))
> +
> +(define (zfs-mcron-auto-snapshot-jobs conf)
> + (let* ((user-auto-snapshot-keep (zfs-configuration-auto-snapshot-keep conf))
> + ;; assoc-ref has earlier entries overriding later ones.
> + (auto-snapshot-keep (append user-auto-snapshot-keep
> + %default-auto-snapshot-keep))
> + (auto-snapshot? (zfs-configuration-auto-snapshot? conf))
> + (zfs-auto-snapshot-package (make-zfs-auto-snapshot-package conf))
> + (zfs-auto-snapshot (file-append zfs-auto-snapshot-package
> + "/sbin/zfs-auto-snapshot")))
> + (map
> + (lambda (label)
> + (let ((keep (assoc-ref auto-snapshot-keep label))
> + (sched (assoc-ref %auto-snapshot-mcron-schedule label)))
> + #~(job '#$sched
> + (string-append #$zfs-auto-snapshot
> + " --quiet --syslog "
> + " --label=" #$(symbol->string label)
> + " --keep=" #$(number->string keep)
> + " //"))))
> + '(frequent hourly daily weekly monthly))))
Maybe use something like
(map first %default-auto-snapshot-keep)
to avoid having to changing it if things change in the future.
> +(define (zfs-mcron-auto-scrub-jobs conf)
> + (let* ((zfs-package (make-zfs-package conf))
> + (zpool (file-append zfs-package "/sbin/zpool"))
> + (auto-scrub (zfs-configuration-auto-scrub conf))
> + (sched (cond
> + ((eq? auto-scrub 'weekly) "0 0 * * 7")
> + ((eq? auto-scrub 'monthly) "0 0 1 * *")
> + (else auto-scrub))))
> + (list
> + #~(job '#$sched
> + ;; Suppress errors: if there are no ZFS pools, the
> + ;; scrub will not be given any arguments, which makes
> + ;; it error out.
> + (string-append "(" #$zpool " scrub `" #$zpool " list -o name -H` "
> + "> /dev/null 2>&1) "
> + "|| exit 0")))))
> +
> +(define (zfs-mcron-jobs conf)
> + (append (zfs-mcron-auto-snapshot-jobs conf)
> + (if (zfs-configuration-auto-scrub conf)
> + (zfs-mcron-auto-scrub-jobs conf)
> + '())))
> +
> +(define zfs-service-type
> + (service-type
> + (name 'zfs)
> + (extensions
> + (list ;; Install OpenZFS kernel module into kernel profile.
> + (service-extension linux-loadable-module-service-type
> + zfs-loadable-modules)
> + ;; And load it.
> + (service-extension kernel-module-loader-service-type
> + (const '("zfs")))
> + ;; Make sure ZFS pools and datasets are mounted at
> + ;; boot.
> + (service-extension shepherd-root-service-type
> + zfs-shepherd-services)
> + ;; Make sure user-processes don't start until
> + ;; after ZFS does.
> + (service-extension user-processes-service-type
> + zfs-user-processes)
> + ;; Install automated scrubbing and snapshotting.
> + (service-extension mcron-service-type
> + zfs-mcron-jobs)
> +
> + ;; Install ZFS management commands in the system
> + ;; profile.
> + (service-extension profile-service-type
> + (compose list make-zfs-package))
> + ;; Install ZFS udev rules.
> + (service-extension udev-service-type
> + (compose list make-zfs-package))))
> + (description "Installs ZFS, an advanced filesystem and volume manager.")))
> --
> 2.31.1
I haven’t tested anything, but the rest looks good! :-)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692]
2021-09-02 21:24 ` [bug#45692] Gaslighting Mason Loring Bliss
2021-09-03 12:22 ` Maxime Devos
@ 2021-09-06 7:59 ` zimoun
1 sibling, 0 replies; 81+ messages in thread
From: zimoun @ 2021-09-06 7:59 UTC (permalink / raw)
To: Mason Loring Bliss; +Cc: Maxime Devos, guix-maintainers, 45692
Hi Mason,
On Thu, 02 Sep 2021 at 17:24, Mason Loring Bliss <mason@blisses.org> wrote:
> Gaslighting in an attempt to sabotage the
> adoption of high-quality free software is somewhat poor form and not at all
> useful.
Using such language (emphasized by the subject), you are not creating a
positive environment. Please be respectful when differing viewpoints
and experiences. And please assume good faith avoiding to jump in
negative opinionated conclusions.
Thanks,
simon
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-02 20:57 ` Maxime Devos
2021-09-02 22:22 ` Maxime Devos
2021-09-04 18:58 ` raid5atemyhomework via Guix-patches via
@ 2021-09-06 8:08 ` zimoun
2021-09-06 10:40 ` Maxime Devos
2 siblings, 1 reply; 81+ messages in thread
From: zimoun @ 2021-09-06 8:08 UTC (permalink / raw)
To: Maxime Devos; +Cc: raid5atemyhomework, 45692@debbugs.gnu.org
Hi Maxime.
On Thu, 02 Sep 2021 at 22:57, Maxime Devos <maximedevos@telenet.be> wrote:
> See also <https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/>.
Reading the Software Freedom Conservancy analysis, from my
understanding, the issue is about distributing the resulting *binary*
Linux+ZFS. Other said, the distribution of the zfs.ko is an issue, not
provide a way to build it locally. Well, it does not appear to be a GPL
violation, IIUC the SFC analysis.
Thanks for taking the time to carefully review all the aspects.
Cheers,
simon
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-06 8:08 ` zimoun
@ 2021-09-06 10:40 ` Maxime Devos
2021-09-06 11:08 ` raid5atemyhomework via Guix-patches via
2021-09-06 17:17 ` zimoun
0 siblings, 2 replies; 81+ messages in thread
From: Maxime Devos @ 2021-09-06 10:40 UTC (permalink / raw)
To: zimoun; +Cc: raid5atemyhomework, 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 2942 bytes --]
zimoun schreef op ma 06-09-2021 om 10:08 [+0200]:
> Hi Maxime.
Hi zimoun,
>
> On Thu, 02 Sep 2021 at 22:57, Maxime Devos <maximedevos@telenet.be> wrote:
>
> > See also <https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/>;.
>
> Reading the Software Freedom Conservancy analysis, from my
> understanding, the issue is about distributing the resulting *binary*
> Linux+ZFS.
Previously I thought that ‘Is The Analysis Different With Source-Only Distribution?’
somehow did not apply here, though I cannot remember the reasoning well (maybe
my reasoning was bogus?).
However, SFC did note in that section:
‘Nevertheless, there may be arguments for contributory and/or indirect copyright
infringement in many jurisdictions. We present no specific analysis ourselves on
the efficacy of a contributory infringement claim regarding source-only distributions
of ZFS and Linux. However, in our GPL litigation experience, we have noticed that
judges are savvy at sniffing out attempts to circumvent legal requirements, and they
are skeptical about attempts to exploit loopholes. Furthermore, we cannot predict
Oracle's view — given its past willingness to enforce copyleft licenses, and Oracle's
recent attempts to adjudicate the limits of copyright in Court. Downstream users should
consider carefully before engaging in even source-only distribution.’
I'm completely unfamiliar with the notion of ‘contributory copyring infringement’
or ‘indirect copyright infringement’. ‘Savvy judges skeptical at attempts to
exploit loopholes’ seems plausible to me in my inexpert opinion.
> Other said, the distribution of the zfs.ko is an issue, not
> provide a way to build it locally. Well, it does not appear to be a GPL
> violation, IIUC the SFC analysis.
I neglected that the terms of the GPL that come into play depend on whether
one is only distributing source code (*) or also binaries, and whether one is
distributing _modified_ source code.
I don't quite see a GPL violation anymore if we only distribute unmodified
source code. However, what about freedom (1) and (3) (freedom to [...] and
change the program in source form and (3) distribute modified versions)?
I was going to write something here about why that would not be (legally)
possible because of the CDDL and GPL, but I forgot why it wouldn't be
possible. Maybe it is actually possible?
(*) raid5atemyhomework noted that guix does _not_ distribute source code,
it only points to source code locations. I don't quite agree. From my
point of view, on whose server the source code is hosted is merely a
technicality, guix is just ‘out-sourcing’ the source code distribution.
Besides, ci.guix.gnu.org keeps a copy of the source code, and
(guix download) will try to download from ci.guix.gnu.org.
> Thanks for taking the time to carefully review all the aspects.
Greetings,
Maxime
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-04 21:19 ` Xinglu Chen
@ 2021-09-06 10:52 ` raid5atemyhomework via Guix-patches via
2021-09-06 14:22 ` Xinglu Chen
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-09-06 10:52 UTC (permalink / raw)
To: Xinglu Chen; +Cc: 45692@debbugs.gnu.org
Hello Xinglu Chen,
Thank you for your interest.
> You might want to bring up the topic of subsystem maintainers on the
> guix-devel mailing list to get some more attention.
Not personally interested.
> > +The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
> > +@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
> > +defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
> > +
> > +@end table
> > +@end deftp
>
> IIUC, there is not way specify ZFS pools in the ‘file-systems’ field of
> an <operating-system> record. Does this mean that ZFS as the root file
> system is not supported, and if so, is there a particular reason for
> this?
No, and as you saw, that requires some more work.
In particular, older versions of this patchset included the ability to add ZFS pools / datasets `file-system` objects, but the exact implementation was objected to, with no suggestion for what alternative to use instead. Since I thought it was contentious, I removed it entirely instead.
Note that ZFS-on-Root on Guix is even harder because of the need to put loading in `initrd`, and a lot of coding as well, not just `file-system`s support.
Given the sheer lack of review and etc etc, I am not encouraged to write more code that will remain unreviewed and unmerged. Maybe if this gets merged as-is, I will, but otherwise, I don't see the point.
> The field name usually just contains the package name, so ‘zfs’ and
> ‘zfs-auto-snapshot’ instead of ‘base-zfs’ and ‘base-zfs-auto-snapshot’.
The point is that the service does **NOT** use `base-zfs` directly --- it creates a new version of that package, targeted towards the specific kernel you provided. This is necessary since internal kernel APIs and ABIs may change between versions, even minor v==ersions (Linux has no commitment to keeping kernel interfaces stable, it only has a commitment to keeping userspace interfaces stable, and OpenZFS **requires** the kernel interfaces, so it is safest to compile specifically to the kernel version that is used).
Thus the ***`base-`*** prefixes: the `zfs-service-type` does not use the `base-zfs` and `base-zfs-autosnapshot` packages as-is, they are instead used as the basis for the actual packages that are compiled and installed into the system. I thought this would be sufficiently different from other services, which use package names as-is (but use the packages as-is, without inheriting from them, unlike this service) that the `base-` prefix was justified.
> > - ;; Output to current-error-port, otherwise the
> >
> >
> > - ;; user will not see any prompts for passwords
> >
> >
> > - ;; of encrypted datasets.
> >
> >
> > - ;; XXX Maybe better to explicitly open /dev/console ?
> >
> >
>
> Seeing this comment, I assume that encrypted pools are supported, right?
Encrypted datasets are supported, yes. If you set `keylocation=prompt` then the `init` process will pause and ask for the password on the console. You can even use `keylocation=file:///some/keyfile`, I also tested that in a VM. My own production setup (which doesn't use this service, but includes some code copy-pasted from this service) uses `keylocation=file:///*elided*`.
> > - '(frequent hourly daily weekly monthly))))
> >
> >
>
> Maybe use something like
>
> (map first %default-auto-snapshot-keep)
>
> to avoid having to changing it if things change in the future.
Good idea, thank you.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-06 10:40 ` Maxime Devos
@ 2021-09-06 11:08 ` raid5atemyhomework via Guix-patches via
2021-09-06 17:17 ` zimoun
1 sibling, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-09-06 11:08 UTC (permalink / raw)
To: Maxime Devos; +Cc: 45692@debbugs.gnu.org, zimoun
Hello Maxime,
> > Other said, the distribution of the zfs.ko is an issue, not
> > provide a way to build it locally. Well, it does not appear to be a GPL
> > violation, IIUC the SFC analysis.
>
> I neglected that the terms of the GPL that come into play depend on whether
> one is only distributing source code () or also binaries, and whether one is
> distributing modified source code.
> I don't quite see a GPL violation anymore if we only distribute unmodified
> source code. However, what about freedom (1) and (3) (freedom to [...] and
> change the program in source form and (3) distribute modified versions)?
> I was going to write something here about why that would not be (legally)
> possible because of the CDDL and GPL, but I forgot why it wouldn't be
> possible. Maybe it is actually possible?
> () raid5atemyhomework noted that guix does not distribute source code,it only points to source code locations. I don't quite agree. From my
> point of view, on whose server the source code is hosted is merely a
> technicality, guix is just ‘out-sourcing’ the source code distribution.
> Besides, ci.guix.gnu.org keeps a copy of the source code, and
> (guix download) will try to download from ci.guix.gnu.org.
*shrug* if so, take note that the `zfs` package modifies the build system and even some of the source code in order to adapt it to the slightly different Guix environment (in particular, Guix does not give an FHS to packages being built, so we need to point some paths directly to absolute paths).
I am not really interested in modifying the patch unless the license issue is resolved, so I will wait until Guix maintainers can actually agree on whether there is a license violation or not.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-06 10:52 ` raid5atemyhomework via Guix-patches via
@ 2021-09-06 14:22 ` Xinglu Chen
0 siblings, 0 replies; 81+ messages in thread
From: Xinglu Chen @ 2021-09-06 14:22 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 4140 bytes --]
On Mon, Sep 06 2021, raid5atemyhomework via Guix-patches via wrote:
> Hello Xinglu Chen,
>
> Thank you for your interest.
You are welcome! Thank you for working on ZFS support!
>> You might want to bring up the topic of subsystem maintainers on the
>> guix-devel mailing list to get some more attention.
>
> Not personally interested.
>
>
>> > +The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
>> > +@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
>> > +defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
>> > +
>> > +@end table
>> > +@end deftp
>>
>> IIUC, there is not way specify ZFS pools in the ‘file-systems’ field of
>> an <operating-system> record. Does this mean that ZFS as the root file
>> system is not supported, and if so, is there a particular reason for
>> this?
>
> No, and as you saw, that requires some more work.
>
> In particular, older versions of this patchset included the ability to
> add ZFS pools / datasets `file-system` objects, but the exact
> implementation was objected to, with no suggestion for what
> alternative to use instead. Since I thought it was contentious, I
> removed it entirely instead.
>
> Note that ZFS-on-Root on Guix is even harder because of the need to
> put loading in `initrd`, and a lot of coding as well, not just
> `file-system`s support.
>
> Given the sheer lack of review and etc etc, I am not encouraged to
> write more code that will remain unreviewed and unmerged. Maybe if
> this gets merged as-is, I will, but otherwise, I don't see the point.
I understand your feeling; hopefully we can get the patch merged soon!
:-)
>> The field name usually just contains the package name, so ‘zfs’ and
>> ‘zfs-auto-snapshot’ instead of ‘base-zfs’ and ‘base-zfs-auto-snapshot’.
>
>
> The point is that the service does **NOT** use `base-zfs` directly ---
> it creates a new version of that package, targeted towards the
> specific kernel you provided. This is necessary since internal kernel
> APIs and ABIs may change between versions, even minor v==ersions
> (Linux has no commitment to keeping kernel interfaces stable, it only
> has a commitment to keeping userspace interfaces stable, and OpenZFS
> **requires** the kernel interfaces, so it is safest to compile
> specifically to the kernel version that is used).
>
> Thus the ***`base-`*** prefixes: the `zfs-service-type` does not use
> the `base-zfs` and `base-zfs-autosnapshot` packages as-is, they are
> instead used as the basis for the actual packages that are compiled
> and installed into the system. I thought this would be sufficiently
> different from other services, which use package names as-is (but use
> the packages as-is, without inheriting from them, unlike this service)
> that the `base-` prefix was justified.
Ah, that makes sense, then I am fine with keeping it as-is.
>> > - ;; Output to current-error-port, otherwise the
>> >
>> >
>> > - ;; user will not see any prompts for passwords
>> >
>> >
>> > - ;; of encrypted datasets.
>> >
>> >
>> > - ;; XXX Maybe better to explicitly open /dev/console ?
>> >
>> >
>>
>> Seeing this comment, I assume that encrypted pools are supported, right?
>
> Encrypted datasets are supported, yes. If you set
> `keylocation=prompt` then the `init` process will pause and ask for
> the password on the console. You can even use
> `keylocation=file:///some/keyfile`, I also tested that in a VM. My
> own production setup (which doesn't use this service, but includes
> some code copy-pasted from this service) uses
> `keylocation=file:///*elided*`.
Cool, hopefully I will get to test this soon. :-)
>> > - '(frequent hourly daily weekly monthly))))
>> >
>> >
>>
>> Maybe use something like
>>
>> (map first %default-auto-snapshot-keep)
>>
>> to avoid having to changing it if things change in the future.
>
> Good idea, thank you.
>
> Thanks
> raid5atemyhomework
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-06 10:40 ` Maxime Devos
2021-09-06 11:08 ` raid5atemyhomework via Guix-patches via
@ 2021-09-06 17:17 ` zimoun
2021-09-07 9:54 ` Maxime Devos
1 sibling, 1 reply; 81+ messages in thread
From: zimoun @ 2021-09-06 17:17 UTC (permalink / raw)
To: Maxime Devos; +Cc: raid5atemyhomework, 45692@debbugs.gnu.org
Hi Maxime,
On Mon, 6 Sept 2021 at 12:41, Maxime Devos <maximedevos@telenet.be> wrote:
> ‘Nevertheless, there may be arguments for contributory and/or indirect copyright
> infringement in many jurisdictions. We present no specific analysis ourselves on
> the efficacy of a contributory infringement claim regarding source-only distributions
> of ZFS and Linux. However, in our GPL litigation experience, we have noticed that
> judges are savvy at sniffing out attempts to circumvent legal requirements, and they
> are skeptical about attempts to exploit loopholes. Furthermore, we cannot predict
> Oracle's view — given its past willingness to enforce copyleft licenses, and Oracle's
> recent attempts to adjudicate the limits of copyright in Court. Downstream users should
> consider carefully before engaging in even source-only distribution.’
The « Nevertheless » is because the previous sentence is:
--8<---------------cut here---------------start------------->8---
[…] Therefore, the analysis is simpler, and we find no specific clause
in either license that prohibits source-only redistribution of Linux and
ZFS, even on the same distribution media.
Nevertheless, […]
--8<---------------cut here---------------end--------------->8---
My understanding is:
- binary distribution violates licenses
- source-only distribution appears to be fine
- SFC cannot guarantee because all the arguments about source-only
distribution have never been tested in Court.
Moreover, they write an explicit paragraph why « You cannot and should
not rely on this document as legal advice. » ;-)
> I don't quite see a GPL violation anymore if we only distribute unmodified
Good. :-)
> source code. However, what about freedom (1) and (3) (freedom to [...] and
> change the program in source form and (3) distribute modified versions)?
Each license is free [1]. Therefore, they respect all the freedoms.
The issue is about linking the result and distribute the binary.
1: <https://directory.fsf.org/wiki/License:CDDL-1.0>
> (*) raid5atemyhomework noted that guix does _not_ distribute source code,
> it only points to source code locations. I don't quite agree. From my
> point of view, on whose server the source code is hosted is merely a
> technicality, guix is just ‘out-sourcing’ the source code distribution.
> Besides, ci.guix.gnu.org keeps a copy of the source code, and
> (guix download) will try to download from ci.guix.gnu.org.
Indeed, ci.guix.gnu.org keeps a copy of (as much as possible) source
code. But ci.guix.gnu.org does not distribute all the corresponding
binaries: see 'arguments' '#:substitutable? #f'. It is already the
case for the package 'zfs':
--8<---------------cut here---------------start------------->8---
(arguments
`(;; The ZFS kernel module should not be downloaded since the license
;; terms don't allow for distributing it, only building it locally.
#:substitutable? #f
--8<---------------cut here---------------end--------------->8---
<https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/file-systems.scm?id=a4ffe3d145b00736f5fdf53ee2c70a7e48592e83#n1175>
As explained in the initial submission [2], this patch set is just a
"glue" usable by the user locally. No binaries on the Guix side is
involved. All the source-code is under free license.
2: <http://issues.guix.gnu.org/45692#0>
Cheers,
simon
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-06 17:17 ` zimoun
@ 2021-09-07 9:54 ` Maxime Devos
2021-09-08 1:23 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: Maxime Devos @ 2021-09-07 9:54 UTC (permalink / raw)
To: zimoun; +Cc: raid5atemyhomework, 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 128 bytes --]
Hi,
To be clear, I don't oppose the inclusion of the ZFS service
on basis of the licensing anymore.
Greetings,
Maxime.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-07 9:54 ` Maxime Devos
@ 2021-09-08 1:23 ` raid5atemyhomework via Guix-patches via
2021-09-15 14:04 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-09-08 1:23 UTC (permalink / raw)
To: Maxime Devos; +Cc: 45692@debbugs.gnu.org, zimoun
Good morning Maxime,
> Hi,
>
> To be clear, I don't oppose the inclusion of the ZFS service
> on basis of the licensing anymore.
Okay. I'm busy right now, so will get around to updating the patch next week.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-08 1:23 ` raid5atemyhomework via Guix-patches via
@ 2021-09-15 14:04 ` raid5atemyhomework via Guix-patches via
2021-09-21 9:42 ` zimoun
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-09-15 14:04 UTC (permalink / raw)
To: Maxime Devos; +Cc: 45692@debbugs.gnu.org, zimoun
hello,
>
> > Hi,
> > To be clear, I don't oppose the inclusion of the ZFS service
> > on basis of the licensing anymore.
>
> Okay. I'm busy right now, so will get around to updating the patch next week.
Scratch that, I'm busy this week as well, maybe next next week.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v4 3/3] gnu: Add ZFS service type.
2021-09-15 14:04 ` raid5atemyhomework via Guix-patches via
@ 2021-09-21 9:42 ` zimoun
0 siblings, 0 replies; 81+ messages in thread
From: zimoun @ 2021-09-21 9:42 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: Maxime Devos, 45692@debbugs.gnu.org
Hi,
On Wed, 15 Sep 2021 at 14:04, raid5atemyhomework <raid5atemyhomework@protonmail.com> wrote:
>> Okay. I'm busy right now, so will get around to updating the patch next week.
>
> Scratch that, I'm busy this week as well, maybe next next week.
Hope that you will find the time next next week. :-)
Just a comment to ease the application of patch. Once rebased on
origin/master, you should create the patch set using:
git format-patch --base=origin/master
because then it ends with this, for instance,
--8<---------------cut here---------------start------------->8---
base-commit: d14221bf65cfbe7f8f5b7cac44132087cab70bf5
--
2.28.0
--8<---------------cut here---------------end--------------->8---
which provide the commit where the patch applies and thus ease work at
review time, IMHO. See details [1].
Cheers,
simon
1: <http://issues.guix.gnu.org/issue/43946>
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type.
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (9 preceding siblings ...)
2021-09-02 21:24 ` [bug#45692] Gaslighting Mason Loring Bliss
@ 2021-09-30 14:56 ` raid5atemyhomework via Guix-patches via
2021-10-19 13:18 ` raid5atemyhomework via Guix-patches via
2022-01-07 4:21 ` [bug#45692] " raid5atemyhomework via Guix-patches via
2022-03-19 14:25 ` [bug#45692] (No Subject) raid5atemyhomework via Guix-patches via
12 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-09-30 14:56 UTC (permalink / raw)
To: zimoun; +Cc: Maxime Devos, 45692@debbugs.gnu.org
Sorry for the lateness everyone.
Hope this one gets reviewed and merged.
--
From 3803e046566278fe12d64f6e39564e9602bf434d Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Thu, 30 Sep 2021 16:58:46 +0800
Subject: [PATCH] gnu: Add ZFS service type.
* gnu/services/file-systems.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
* gnu/services/base.scm: Export dependency->shepherd-service-name.
* doc/guix.texi (ZFS File System): New subsection.
---
doc/guix.texi | 351 ++++++++++++++++++++++++++++++++
gnu/local.mk | 2 +
gnu/services/base.scm | 4 +-
gnu/services/file-systems.scm | 363 ++++++++++++++++++++++++++++++++++
4 files changed, 719 insertions(+), 1 deletion(-)
create mode 100644 gnu/services/file-systems.scm
diff --git a/doc/guix.texi b/doc/guix.texi
index a72a726b54..dd38103953 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -97,6 +97,7 @@ Copyright @copyright{} 2021 Hui Lu@*
Copyright @copyright{} 2021 pukkamustard@*
Copyright @copyright{} 2021 Alice Brenon@*
Copyright @copyright{} 2021 Andrew Tropin@*
+Copyright @copyright{} 2021 raid5atemyhomework@*
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -14435,6 +14436,356 @@ a file system declaration such as:
compress-force=zstd,space_cache=v2"))
@end lisp
+
+@node ZFS File System
+@subsection ZFS File System
+
+Support for ZFS file systems in Guix is based on the OpenZFS project.
+OpenZFS currently only supports Linux-Libre and is not available on the
+Hurd.
+
+OpenZFS is free software; unfortunately its license is incompatible with
+the GNU General Public License (GPL), the license of the Linux kernel,
+which means they cannot be distributed together. However, as a user,
+you can choose to build ZFS and use it together with Linux; you can
+even rely on Guix to automate this task. See
+@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
+the Free Software Foundation} for more information.
+
+As a large and complex kernel module, OpenZFS has to be compiled for a
+specific version of Linux-Libre. At times, the latest OpenZFS package
+available in Guix is not compatible with the latest Linux-Libre version.
+Thus, directly installing the @code{zfs} package can fail.
+
+Instead, you are recommended to select a specific older long-term-support
+Linux-Libre kernel. Do not use @code{linux-libre-lts}, as even the
+latest long-term-support kernel may be too new for @code{zfs}. Instead,
+explicitly select a specific older version, such as @code{linux-libre-5.10},
+and upgrade it manually later as new long-term-support kernels become
+available that you have confirmed is compatible with the latest available
+OpenZFS version on Guix.
+
+For example, you can modify your system configuration file to a specific
+Linux-Libre version and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+ #;@dots{}
+ linux)
+(use-service-modules
+ #;@dots{}
+ file-systems)
+
+(define my-kernel linux-libre-5.10)
+
+(operating-system
+ (kernel my-kernel)
+ #;@dots{}
+ (services
+ (cons* (service zfs-service-type
+ (zfs-configuration
+ (kernel my-kernel)))
+ #;@dots{}
+ %desktop-services))
+ #;@dots{})
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type for a service that adds ZFS support to your operating
+system. The service is configured using a @code{zfs-configuration}
+record.
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+ (zfs-configuration
+ (kernel linux-libre-5.4)))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS support in Guix
+System. Its fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux-Libre kernel to compile OpenZFS for. This field
+is always required. It @emph{must} be the same kernel you use in your
+@code{operating-system} form.
+
+@item @code{base-zfs} (default: @code{zfs})
+The OpenZFS package that will be compiled for the given Linux-Libre kernel.
+
+@item @code{base-zfs-auto-snapshot} (default: @code{zfs-auto-snapshot})
+The @code{zfs-auto-snapshot} package to use. It will be modified to
+specifically use the OpenZFS compiled for your kernel.
+
+@item @code{dependencies} (default: @code{'()})
+A list of @code{<file-system>} or @code{<mapped-device>} records that must
+be mounted or opened before OpenZFS scans for pools to import. For example,
+if you have set up LUKS containers as leaf VDEVs in a pool, you have to
+include their corresponding @code{<mapped-ddevice>} records so that OpenZFS
+can import the pool correctly at bootup.
+
+@item @code{auto-mount?} (default: @code{#t})
+Whether to mount datasets with the ZFS @code{mountpoint} property automatically
+at startup. This is the behavior that ZFS users usually expect. You might
+set this to @code{#f} for an operating system intended as a ``rescue'' system
+that is intended to help debug problems with the disks rather than actually
+work in production.
+
+@item @code{auto-scrub} (default: @code{'weekly})
+Specifies how often to scrub all pools. Can be the symbols @code{'weekly} or
+@code{'monthly}, or a schedule specification understood by
+@xref{mcron, mcron job specifications,, mcron, GNU@tie{}mcron}, such as
+@code{"0 3 * * 6"} for ``every 3AM on Saturday''.
+It can also be @code{#f} to disable auto-scrubbing (@strong{not recommended}).
+
+The general guideline is to scrub weekly when using consumer-quality drives, and
+to scrub monthly when using enterprise-quality drives.
+
+@code{'weekly} scrubs are done on Sunday midnight, while @code{monthly} scrubs
+are done on midnight on the first day of each month.
+
+@item @code{auto-snapshot?} (default: @code{#t})
+Specifies whether to auto-snapshot by default. If @code{#t}, then snapshots
+are automatically created except for ZFS datasets with the
+@code{com.sun:auto-snapshot} ZFS vendor property set to @code{false}.
+
+If @code{#f}, snapshots will not be automatically created, unless the ZFS
+dataset has the @code{com.sun:auto-snapshot} ZFS vendor property set to
+@code{true}.
+
+@item @code{auto-snapshot-keep} (default: @code{'()})
+Specifies an association list of symbol-number pairs, indicating the number
+of automatically-created snapshots to retain for each frequency type.
+
+If not specified via this field, by default there are 4 @code{frequent}, 24
+@code{hourly}, 31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly} snapshots.
+
+For example:
+
+@lisp
+(zfs-configuration
+ (kernel my-kernel)
+ (auto-snapshot-keep
+ '((frequent . 8)
+ (hourly . 12))))
+@end lisp
+
+The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
+@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
+defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
+
+@end table
+@end deftp
+
+@subsubsection ZFS Auto-Snapshot
+
+The ZFS service on Guix System supports auto-snapshots as implemented in the
+Solaris operating system.
+
+@code{frequent} (every 15 minutes), @code{hourly}, @code{daily}, @code{weekly},
+and @code{monthly} snapshots are created automatically for ZFS datasets that
+have auto-snapshot enabled. They will be named, for example,
+@code{zfs-auto-snap_frequent-2021-03-22-1415}. You can continue to use
+manually-created snapshots as long as they do not conflict with the naming
+convention used by auto-snapshot. You can also safely manually destroy
+automatically-created snapshots, for example to free up space.
+
+The @code{com.sun:auto-snapshot} ZFS property controls auto-snapshot on a
+per-dataset level. Sub-datasets will inherit this property from their parent
+dataset, but can have their own property.
+
+You @emph{must} set this property to @code{true} or @code{false} exactly,
+otherwise it will be treated as if the property is unset.
+
+For example:
+
+@example
+# zfs list -o name
+NAME
+tank
+tank/important-data
+tank/tmp
+# zfs set com.sun:auto-snapshot=true tank
+# zfs set com.sun:auto-snapshot=false tank/tmp
+@end example
+
+The above will set @code{tank} and @code{tank/important-data} to be
+auto-snapshot, while @code{tank/tmp} will not be auto-snapshot.
+
+If the @code{com.sun:auto-snapshot} property is not set for a dataset
+(the default when pools and datasets are created), then whether
+auto-snapshot is done or not will depend on the @code{auto-snapshot?}
+field of the @code{zfs-configuration} record.
+
+There are also @code{com.sun:auto-snapshot:frequent},
+@code{com.sun:auto-snapshot:hourly}, @code{com.sun:auto-snapshot:daily},
+@code{com.sun:auto-snapshot:weekly}, and @code{com.sun:auto-snapshot:monthly}
+properties that give finer-grained control of whether to auto-snapshot a
+dataset at a particular schedule.
+
+The number of snapshots kept for all datasets can be overridden via the
+@code{auto-snapshot-keep} field of the @code{zfs-configuration} record.
+There is currently no support to have different numbers of snapshots to
+keep for different datasets.
+
+@subsubsection ZVOLs
+
+ZFS supports ZVOLs, block devices that ZFS exposes to the operating
+system in the @code{/dev/zvol/} directory. The ZVOL will have the same
+resilience and self-healing properties as other datasets on your ZFS pool.
+ZVOLs can also be snapshotted (and will be included in auto-snapshotting
+if enabled), which snapshots the state of the block device, effectively
+snapshotting the hosted file system.
+
+You can put any file system inside the ZVOL. However, in order to mount this
+file system at system start, you need to add @code{%zfs-zvol-dependency} as a
+dependency of each file system inside a ZVOL.
+
+@defvr {Scheme Variable} %zfs-zvol-dependency
+An artificial @code{<mapped-device>} which tells the file system mounting
+service to wait for ZFS to provide ZVOLs before mounting the
+@code{<file-system>} dependent on it.
+@end defvr
+
+For example, suppose you create a ZVOL and put an ext4 filesystem
+inside it:
+
+@example
+# zfs create -V 100G tank/ext4-on-zfs
+# mkfs.ext4 /dev/zvol/tank/ext4-on-zfs
+# mkdir /ext4-on-zfs
+# mount /dev/zvol/tank/ext4-on-zfs /ext4-on-zfs
+@end example
+
+You can then set this up to be mounted at boot by adding this to the
+@code{file-systems} field of your @code{operating-system} record:
+
+@lisp
+(file-system
+ (device "/dev/zvol/tank/ext4-on-zfs")
+ (mount-point "/ext4-on-zfs")
+ (type "ext4")
+ (dependencies (list %zfs-zvol-dependency)))
+@end lisp
+
+You @emph{must not} add @code{%zfs-zvol-dependency} to your
+@code{operating-system}'s @code{mapped-devices} field, and you @emph{must
+not} add it (or any @code{<file-system>}s dependent on it) to the
+@code{dependencies} field of @code{zfs-configuration}. Finally, you
+@emph{must not} use @code{%zfs-zvol-dependency} unless you actually
+instantiate @code{zfs-service-type} on your system.
+
+@subsubsection Unsupported Features
+
+Some common features and uses of ZFS are currently not supported, or not
+fully supported, on Guix.
+
+@enumerate
+@item
+Shepherd-managed daemons that are configured to read from or write to ZFS
+mountpoints need to include @code{user-processes} in their @code{requirement}
+field. This is the earliest that ZFS file systems are assured of being
+mounted.
+
+Generally, most daemons will, directly or indirectly, require
+@code{networking}, or @code{user-processes}, or both. Most implementations
+of @code{networking} will also require @code{user-processes} so daemons that
+require only @code{networking} will also generally start up after
+@code{user-processes}. A notable exception, however, is
+@code{static-networking-service-type}. You will need to explicitly add
+@code{user-processes} as a @code{requirement} of your @code{static-networking}
+record.
+
+@item
+@code{mountpoint=legacy} ZFS file systems. The handlers for the Guix mounting
+system have not yet been modified to support ZFS, and will expect @code{/dev}
+paths in the @code{<file-system>}'s @code{device} field, but ZFS file systems
+are referred to via non-path @code{pool/file/system} names. Such file systems
+also need to be mounted @emph{after} OpenZFS has scanned for pools.
+
+You can still manually mount these file systems after system boot; what is
+only unsupported is mounting them automatically at system boot by specifying
+them in @code{<file-system>} records of your @code{operating-system}.
+
+@item
+@code{/home} on ZFS. Guix will create home directories for users, but this
+process currently cannot be scheduled after ZFS file systems are mounted.
+Thus, the ZFS file system might be mounted @emph{after} Guix has created
+home directories at boot, at which point OpenZFS will refuse to mount since
+the mountpoint is not empty. However, you @emph{can} create an ext4, xfs,
+btrfs, or other supported file system inside a ZVOL, have that depend on
+@code{%zfs-zvol-dependency}, and set it to mount on the @code{/home}
+directory; they will be scheduled to mount before the @code{user-homes}
+process.
+
+Similarly, other locations like @code{/var}, @code{/gnu/store} and so
+on cannot be reliably put in a ZFS file system, though they may be
+possible to create as other file systems inside ZVOL containers.
+
+@item
+@code{/} and @code{/boot} on ZFS. These require Guix to expose more of
+the @code{initrd} very early boot process to services. It also requires
+Guix to have the ability to explicitly load modules while still in
+@code{initrd} (currently kernel modules loaded by
+@code{kernel-module-loader-service-type} are loaded after @code{/} is
+mounted). Further, since one of ZFS's main advantages is that it can
+continue working despite the loss of one or more devices, it makes sense
+to also support installing the bootloader on all devices of the pool that
+contains the @code{/} and @code{/boot}; after all, if ZFS can survive the
+loss of one device, the bootloader should also be able to survive the loss
+of one device.
+
+@item
+ZVOL swap devices. Mapped swap devices need to be listed in
+@code{mapped-devices} to ensure they are opened before the system attempts
+to use them, but you cannot currently add @code{%zfs-zvol-dependency} to
+@code{mapped-devices}.
+
+This will also require significant amounts of testing, as various kernel
+build options and patches may affect how swapping works, which are possibly
+different on Guix System compared to other distributions that this feature is
+known to work on.
+
+@item
+ZFS Event Daemon. Support for this has not been written yet, patches are
+welcome. The main issue is how to design this in a Guix style while
+supporting legacy shell-script styles as well. In particular, OpenZFS itself
+comes with a number of shell scripts intended for ZFS Event Daemon, and we
+need to figure out how the user can choose to use or not use the provided
+scripts (and configure any settings they have) or override with their own
+custom code (which could be shell scripts they have written and trusted from
+previous ZFS installations).
+
+As-is, you can create your own service that activates the ZFS Event Daemon
+by creating the @file{/etc/zfs/zed} directory and filling it appropriately,
+then launching @code{zed}.
+
+@item
+@file{/etc/zfs/zpool.cache}. Currently the ZFS support on Guix always forces
+scanning of all devices at bootup to look for ZFS pools. For systems with
+dozens or hundreds of storage devices, this can lead to slow bootup. One issue
+is that tools should really not write to @code{/etc} which is supposed to be for
+configuration; possibly it could be moved to @code{/var} instead. Another issue
+is that if Guix ever supports @code{/} on ZFS, we would need to somehow keep the
+@code{zpool.cache} file inside the @code{initrd} up-to-date with what is in the
+@code{/} mount point.
+
+@item
+@code{zfs share}. This will require some (unknown amount of) work to integrate
+into the Samba and NFS services of Guix. You @emph{can} manually set up Samba
+and NFS to share any mounted ZFS datasets by setting up their configurations
+properly; it just can't be done for you by @code{zfs share} and the
+@code{sharesmb} and @code{sharenfs} properties.
+@end enumerate
+
+Hopefully, support for the above only requires code to be written, and users
+are encouraged to hack on Guix to implement the above features.
+
@node Mapped Devices
@section Mapped Devices
diff --git a/gnu/local.mk b/gnu/local.mk
index d415b892e9..4147badd49 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -45,6 +45,7 @@
# Copyright © 2021 Sharlatan Hellseher <sharlatanus@gmail.com>
# Copyright © 2021 Dmitry Polyakov <polyakov@liltechdude.xyz>
# Copyright © 2021 Andrew Tropin <andrew@trop.in>
+# Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
#
# This file is part of GNU Guix.
#
@@ -633,6 +634,7 @@ GNU_SYSTEM_MODULES = \
%D%/services/docker.scm \
%D%/services/authentication.scm \
%D%/services/file-sharing.scm \
+ %D%/services/file-systems.scm \
%D%/services/games.scm \
%D%/services/ganeti.scm \
%D%/services/getmail.scm \
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 50865055fe..d5d33aeada 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -186,7 +186,9 @@
references-file
- %base-services))
+ %base-services
+
+ dependency->shepherd-service-name))
;;; Commentary:
;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..867349c3a5
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,363 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+ #:use-module (gnu packages file-systems)
+ #:use-module (gnu services)
+ #:use-module (gnu services base)
+ #:use-module (gnu services linux)
+ #:use-module (gnu services mcron)
+ #:use-module (gnu services shepherd)
+ #:use-module (gnu system mapped-devices)
+ #:use-module (guix gexp)
+ #:use-module (guix modules)
+ #:use-module (guix packages)
+ #:use-module (guix records)
+ #:use-module (srfi srfi-1)
+ #:export (zfs-service-type
+
+ zfs-configuration
+ zfs-configuration?
+ zfs-configuration-kernel
+ zfs-configuration-base-zfs
+ zfs-configuration-base-zfs-auto-snapshot
+ zfs-configuration-dependencies
+ zfs-configuration-auto-mount?
+ zfs-configuration-auto-scrub
+ zfs-configuration-auto-snapshot?
+ zfs-configuration-auto-snapshot-keep
+
+ %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+ zfs-configuration
+ make-zfs-configuration
+ zfs-configuration?
+
+ ;; linux-libre kernel you want to compile the base-zfs module for.
+ (kernel zfs-configuration-kernel)
+
+ ;; the OpenZFS package that will be modified to compile for the
+ ;; given kernel.
+ ;; Because it is modified and not the actual package that is used,
+ ;; we prepend the name 'base-'.
+ (base-zfs zfs-configuration-base-zfs
+ (default zfs))
+
+ ;; the zfs-auto-snapshot package that will be modified to compile
+ ;; for the given kernel.
+ ;; Because it is modified and not the actual package that is used,
+ ;; we prepend the name 'base-'.
+ (base-zfs-auto-snapshot zfs-configuration-base-zfs-auto-snapshot
+ (default zfs-auto-snapshot))
+
+ ;; list of <mapped-device> or <file-system> objects that must be
+ ;; opened/mounted before we import any ZFS pools.
+ (dependencies zfs-configuration-dependencies
+ (default '()))
+
+ ;; #t to mount all mountable datasets by default.
+ ;; #f if not mounting.
+ ;; #t is the expected behavior on other operating systems, the
+ ;; #f is only supported for "rescue" operating systems where
+ ;; the user wants lower-level control of when to mount.
+ (auto-mount? zfs-configuration-auto-mount?
+ (default #t))
+
+ ;; 'weekly for weekly scrubbing, 'monthly for monthly scrubbing, an
+ ;; mcron time specification that can be given to `job`, or #f to
+ ;; disable.
+ (auto-scrub zfs-configuration-auto-scrub
+ (default 'weekly))
+
+ ;; #t to auto-snapshot by default (and `com.sun:auto-snapshot=false`
+ ;; disables auto-snapshot per dataset), #f to not auto-snapshot
+ ;; by default (and `com.sun:auto-snapshot=true` enables auto-snapshot
+ ;; per dataset).
+ (auto-snapshot? zfs-configuration-auto-snapshot?
+ (default #t))
+
+ ;; association list of symbol-number pairs to indicate the number
+ ;; of automatic snapshots to keep for each of 'frequent, 'hourly,
+ ;; 'daily, 'weekly, and 'monthly.
+ ;; e.g. '((frequent . 8) (hourly . 12))
+ (auto-snapshot-keep zfs-configuration-auto-snapshot-keep
+ (default '())))
+
+(define %default-auto-snapshot-keep
+ '((frequent . 4)
+ (hourly . 24)
+ (daily . 31)
+ (weekly . 8)
+ (monthly . 12)))
+
+(define %auto-snapshot-mcron-schedule
+ '((frequent . "0,15,30,45 * * * *")
+ (hourly . "0 * * * *")
+ (daily . "0 0 * * *")
+ (weekly . "0 0 * * 7")
+ (monthly . "0 0 1 * *")))
+
+;; A synthetic and unusable MAPPED-DEVICE intended for use when
+;; the user has created a mountable filesystem inside a ZFS
+;; zvol and wants it mounted inside the configuration.scm.
+(define %zfs-zvol-dependency
+ (mapped-device
+ (source '())
+ (targets '("zvol/*"))
+ (type #f)))
+
+(define (make-zfs-package conf)
+ "Creates a zfs package based on the given zfs-configuration.
+
+ OpenZFS is a kernel package and to ensure best compatibility
+ it should be compiled with the specific Linux-Libre kernel
+ used on the system. This simply overrides the kernel used
+ in compilation with that given in the configuration, which
+ the user has to ensure is the same as in the operating-system."
+ (let ((kernel (zfs-configuration-kernel conf))
+ (base-zfs (zfs-configuration-base-zfs conf)))
+ (package
+ (inherit base-zfs)
+ (arguments (cons* #:linux kernel
+ (package-arguments base-zfs))))))
+
+(define (make-zfs-auto-snapshot-package conf)
+ "Creates a zfs-auto-snapshot package based on the given
+ zfs-configuration.
+
+ Since the OpenZFS tools above are compiled to a specific
+ kernel version, zfs-auto-snapshot --- which calls into the
+ OpenZFS tools --- has to be compiled with the specific
+ modified OpenZFS package created in the make-zfs-package
+ procedure."
+ (let ((zfs (make-zfs-package conf))
+ (base-zfs-auto-snapshot (zfs-configuration-base-zfs-auto-snapshot conf)))
+ (package
+ (inherit base-zfs-auto-snapshot)
+ (inputs `(("zfs" ,zfs))))))
+
+(define (zfs-loadable-modules conf)
+ "Specifies that the specific 'module' output of the OpenZFS
+ package is to be used; for use in indicating it as a
+ loadable kernel module."
+ (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+ "Constructs a list of Shepherd services that is installed
+ by the ZFS Guix service.
+
+ 'zfs-scan' scans all devices for ZFS pools, and makes them
+ available to 'zpool' commands.
+ 'device-mapping-zvol/*' waits for /dev/zvol/* to be
+ populated by 'udev', and runs after 'zfs-scan'.
+ 'zfs-auto-mount' mounts all ZFS datasets with a 'mount'
+ property, which defaults to '/' followed by the name of
+ the dataset.
+
+ All the above behavior is expected by ZFS users from
+ typical ZFS installations. A mild difference is that
+ scanning is usually based on '/etc/zfs/zpool.cache'
+ instead of the 'scan all devices' used below, but that
+ file is questionable in Guix since ideally '/etc/'
+ files are modified by the sysad directly;
+ '/etc/zfs/zpool.cache' is modified by ZFS tools."
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (zfs (file-append zfs-package "/sbin/zfs"))
+ (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
+ (scheme-modules `((srfi srfi-1)
+ (srfi srfi-34)
+ (srfi srfi-35)
+ (rnrs io ports)
+ ,@%default-modules)))
+ (define zfs-scan
+ (shepherd-service
+ (provision '(zfs-scan))
+ (requirement `(root-file-system
+ kernel-module-loader
+ udev
+ ,@(map dependency->shepherd-service-name
+ (zfs-configuration-dependencies conf))))
+ (documentation "Scans for and imports ZFS pools.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error importing pools: ~s~%"
+ (condition-message c))
+ #f))
+ ;; TODO: optionally use a cachefile.
+ (invoke #$zpool "import" "-a" "-N"))))
+ ;; Why not one-shot? Because we don't really want to rescan
+ ;; this each time a requiring process is restarted, as scanning
+ ;; can take a long time and a lot of I/O.
+ (stop #~(const #f))))
+
+ (define device-mapping-zvol/*
+ (shepherd-service
+ (provision '(device-mapping-zvol/*))
+ (requirement '(zfs-scan))
+ (documentation "Waits for all ZFS ZVOLs to be opened.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error opening zvols: ~s~%"
+ (condition-message c))
+ #f))
+ (invoke #$zvol_wait))))
+ (stop #~(const #f))))
+
+ (define zfs-auto-mount
+ (shepherd-service
+ (provision '(zfs-auto-mount))
+ (requirement '(zfs-scan))
+ (documentation "Mounts all non-legacy mounted ZFS filesystems.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error mounting file systems: ~s~%"
+ (condition-message c))
+ #f))
+ ;; Output to current-error-port, otherwise the
+ ;; user will not see any prompts for passwords
+ ;; of encrypted datasets.
+ ;; XXX Maybe better to explicitly open /dev/console ?
+ (with-output-to-port (current-error-port)
+ (lambda ()
+ (invoke #$zfs "mount" "-a" "-l"))))))
+ (stop #~(lambda _
+ ;; Make sure that Shepherd does not have a CWD that
+ ;; is a mounted ZFS filesystem, which would prevent
+ ;; unmounting.
+ (chdir "/")
+ (invoke #$zfs "unmount" "-a" "-f")))))
+
+ `(,zfs-scan
+ ,device-mapping-zvol/*
+ ,@(if (zfs-configuration-auto-mount? conf)
+ `(,zfs-auto-mount)
+ '()))))
+
+(define (zfs-user-processes conf)
+ "Provides the last Shepherd service that 'user-processes' has to
+ wait for.
+
+ If not auto-mounting, then user-processes should only wait for
+ the device scan."
+ (if (zfs-configuration-auto-mount? conf)
+ '(zfs-auto-mount)
+ '(zfs-scan)))
+
+(define (zfs-mcron-auto-snapshot-jobs conf)
+ "Creates a list of mcron jobs for auto-snapshotting, one for each
+ of the standard durations."
+ (let* ((user-auto-snapshot-keep (zfs-configuration-auto-snapshot-keep conf))
+ ;; assoc-ref has earlier entries overriding later ones.
+ (auto-snapshot-keep (append user-auto-snapshot-keep
+ %default-auto-snapshot-keep))
+ (auto-snapshot? (zfs-configuration-auto-snapshot? conf))
+ (zfs-auto-snapshot-package (make-zfs-auto-snapshot-package conf))
+ (zfs-auto-snapshot (file-append zfs-auto-snapshot-package
+ "/sbin/zfs-auto-snapshot")))
+ (map
+ (lambda (label)
+ (let ((keep (assoc-ref auto-snapshot-keep label))
+ (sched (assoc-ref %auto-snapshot-mcron-schedule label)))
+ #~(job '#$sched
+ (lambda ()
+ (system* #$zfs-auto-snapshot
+ "--quiet"
+ "--syslog"
+ #$(string-append "--label="
+ (symbol->string label))
+ #$(string-append "--keep="
+ (number->string keep))
+ "//")))))
+ (map first %auto-snapshot-mcron-schedule))))
+
+(define (zfs-mcron-auto-scrub-jobs conf)
+ "Creates a list of mcron jobs for auto-scrubbing."
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (auto-scrub (zfs-configuration-auto-scrub conf))
+ (sched (cond
+ ((eq? auto-scrub 'weekly) "0 0 * * 7")
+ ((eq? auto-scrub 'monthly) "0 0 1 * *")
+ (else auto-scrub))))
+ (define code
+ ;; We need to get access to (guix build utils) for the
+ ;; invoke procedures.
+ (with-imported-modules (source-module-closure '((guix build utils)))
+ #~(begin
+ (use-modules (guix build utils)
+ (ice-9 ports))
+ ;; The ZFS pools in the system.
+ (define pools
+ (invoke/quiet #$zpool "list" "-o" "name" "-H"))
+ ;; Only scrub if there are actual ZFS pools, as the
+ ;; zpool scrub command errors out if given an empty
+ ;; argument list.
+ (unless (null? pools)
+ ;; zpool scrub only initiates the scrub and otherwise
+ ;; prints nothing. Results are always seen on the
+ ;; zpool status command.
+ (apply invoke #$zpool "scrub" pools)))))
+ (list
+ #~(job '#$sched
+ #$(program-file "mcron-zfs-scrub.scm" code)))))
+
+(define (zfs-mcron-jobs conf)
+ "Creates a list of mcron jobs for ZFS management."
+ (append (zfs-mcron-auto-snapshot-jobs conf)
+ (if (zfs-configuration-auto-scrub conf)
+ (zfs-mcron-auto-scrub-jobs conf)
+ '())))
+
+(define zfs-service-type
+ (service-type
+ (name 'zfs)
+ (extensions
+ (list ;; Install OpenZFS kernel module into kernel profile.
+ (service-extension linux-loadable-module-service-type
+ zfs-loadable-modules)
+ ;; And load it.
+ (service-extension kernel-module-loader-service-type
+ (const '("zfs")))
+ ;; Make sure ZFS pools and datasets are mounted at
+ ;; boot.
+ (service-extension shepherd-root-service-type
+ zfs-shepherd-services)
+ ;; Make sure user-processes don't start until
+ ;; after ZFS does.
+ (service-extension user-processes-service-type
+ zfs-user-processes)
+ ;; Install automated scrubbing and snapshotting.
+ (service-extension mcron-service-type
+ zfs-mcron-jobs)
+
+ ;; Install ZFS management commands in the system
+ ;; profile.
+ (service-extension profile-service-type
+ (compose list make-zfs-package))
+ ;; Install ZFS udev rules.
+ (service-extension udev-service-type
+ (compose list make-zfs-package))))
+ (description "Installs ZFS, an advanced filesystem and volume manager.")))
base-commit: a939011b58c65f4192a10cde9e925e85702bacf4
--
2.33.0
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type.
2021-09-30 14:56 ` [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type raid5atemyhomework via Guix-patches via
@ 2021-10-19 13:18 ` raid5atemyhomework via Guix-patches via
2021-10-27 7:30 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-10-19 13:18 UTC (permalink / raw)
To: zimoun; +Cc: Maxime Devos, 45692@debbugs.gnu.org
***BUMP***
> Sorry for the lateness everyone.
>
> Hope this one gets reviewed and merged.
>
> --------------------------------------------------------------------------
>
> From 3803e046566278fe12d64f6e39564e9602bf434d Mon Sep 17 00:00:00 2001
> From: raid5atemyhomework raid5atemyhomework@protonmail.com
> Date: Thu, 30 Sep 2021 16:58:46 +0800
> Subject: [PATCH] gnu: Add ZFS service type.
>
> - gnu/services/file-systems.scm: New file.
>
> - gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
>
> - gnu/services/base.scm: Export dependency->shepherd-service-name.
>
> - doc/guix.texi (ZFS File System): New subsection.
>
>
> doc/guix.texi | 351 ++++++++++++++++++++++++++++++++
> gnu/local.mk | 2 +
> gnu/services/base.scm | 4 +-
> gnu/services/file-systems.scm | 363 ++++++++++++++++++++++++++++++++++
> 4 files changed, 719 insertions(+), 1 deletion(-)
> create mode 100644 gnu/services/file-systems.scm
>
> diff --git a/doc/guix.texi b/doc/guix.texi
> index a72a726b54..dd38103953 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -97,6 +97,7 @@ Copyright @copyright{} 2021 Hui Lu@*
> Copyright @copyright{} 2021 pukkamustard@*
> Copyright @copyright{} 2021 Alice Brenon@*
> Copyright @copyright{} 2021 Andrew Tropin@*
> +Copyright @copyright{} 2021 raid5atemyhomework@*
>
> Permission is granted to copy, distribute and/or modify this document
> under the terms of the GNU Free Documentation License, Version 1.3 or
> @@ -14435,6 +14436,356 @@ a file system declaration such as:
> compress-force=zstd,space_cache=v2"))
> @end lisp
>
> +
> +@node ZFS File System
> +@subsection ZFS File System
> +
> +Support for ZFS file systems in Guix is based on the OpenZFS project.
> +OpenZFS currently only supports Linux-Libre and is not available on the
> +Hurd.
> +
> +OpenZFS is free software; unfortunately its license is incompatible with
> +the GNU General Public License (GPL), the license of the Linux kernel,
> +which means they cannot be distributed together. However, as a user,
> +you can choose to build ZFS and use it together with Linux; you can
> +even rely on Guix to automate this task. See
> +@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
> +the Free Software Foundation} for more information.
> +
> +As a large and complex kernel module, OpenZFS has to be compiled for a
> +specific version of Linux-Libre. At times, the latest OpenZFS package
> +available in Guix is not compatible with the latest Linux-Libre version.
> +Thus, directly installing the @code{zfs} package can fail.
> +
> +Instead, you are recommended to select a specific older long-term-support
> +Linux-Libre kernel. Do not use @code{linux-libre-lts}, as even the
> +latest long-term-support kernel may be too new for @code{zfs}. Instead,
> +explicitly select a specific older version, such as @code{linux-libre-5.10},
> +and upgrade it manually later as new long-term-support kernels become
> +available that you have confirmed is compatible with the latest available
> +OpenZFS version on Guix.
> +
> +For example, you can modify your system configuration file to a specific
> +Linux-Libre version and add the @code{zfs-service-type} service.
> +
> +@lisp
> +(use-modules (gnu))
> +(use-package-modules
>
> - #;@dots{}
> - linux)
> +(use-service-modules
>
> - #;@dots{}
> - file-systems)
> -
>
> +(define my-kernel linux-libre-5.10)
> +
> +(operating-system
>
> - (kernel my-kernel)
> - #;@dots{}
> - (services
> - (cons* (service zfs-service-type
> - (zfs-configuration
>
>
> - (kernel my-kernel)))
>
>
> - #;@dots{}
>
>
> - %desktop-services))
>
>
> - #;@dots{})
> +@end lisp
>
> -
>
> +@defvr {Scheme Variable} zfs-service-type
> +This is the type for a service that adds ZFS support to your operating
> +system. The service is configured using a @code{zfs-configuration}
> +record.
> +
> +Here is an example use:
> +
> +@lisp
> +(service zfs-service-type
>
> - (zfs-configuration
> - (kernel linux-libre-5.4)))
> +@end lisp
> +@end defvr
>
> -
>
> +@deftp {Data Type} zfs-configuration
> +This data type represents the configuration of the ZFS support in Guix
> +System. Its fields are:
> +
> +@table @asis
> +@item @code{kernel}
> +The package of the Linux-Libre kernel to compile OpenZFS for. This field
> +is always required. It @emph{must} be the same kernel you use in your
> +@code{operating-system} form.
> +
> +@item @code{base-zfs} (default: @code{zfs})
> +The OpenZFS package that will be compiled for the given Linux-Libre kernel.
> +
> +@item @code{base-zfs-auto-snapshot} (default: @code{zfs-auto-snapshot})
> +The @code{zfs-auto-snapshot} package to use. It will be modified to
> +specifically use the OpenZFS compiled for your kernel.
> +
> +@item @code{dependencies} (default: @code{'()})
> +A list of @code{<file-system>} or @code{<mapped-device>} records that must
> +be mounted or opened before OpenZFS scans for pools to import. For example,
> +if you have set up LUKS containers as leaf VDEVs in a pool, you have to
> +include their corresponding @code{<mapped-ddevice>} records so that OpenZFS
> +can import the pool correctly at bootup.
> +
> +@item @code{auto-mount?} (default: @code{#t})
> +Whether to mount datasets with the ZFS @code{mountpoint} property automatically
> +at startup. This is the behavior that ZFS users usually expect. You might
> +set this to @code{#f} for an operating system intended as a `rescue'' system +that is intended to help debug problems with the disks rather than actually +work in production. + +@item @code{auto-scrub} (default: @code{'weekly}) +Specifies how often to scrub all pools. Can be the symbols @code{'weekly} or +@code{'monthly}, or a schedule specification understood by +@xref{mcron, mcron job specifications,, mcron, GNU@tie{}mcron}, such as +@code{"0 3 * * 6"} for`every 3AM on Saturday''.
> +It can also be @code{#f} to disable auto-scrubbing (@strong{not recommended}).
> +
> +The general guideline is to scrub weekly when using consumer-quality drives, and
> +to scrub monthly when using enterprise-quality drives.
> +
> +@code{'weekly} scrubs are done on Sunday midnight, while @code{monthly} scrubs
> +are done on midnight on the first day of each month.
> +
> +@item @code{auto-snapshot?} (default: @code{#t})
> +Specifies whether to auto-snapshot by default. If @code{#t}, then snapshots
> +are automatically created except for ZFS datasets with the
> +@code{com.sun:auto-snapshot} ZFS vendor property set to @code{false}.
> +
> +If @code{#f}, snapshots will not be automatically created, unless the ZFS
> +dataset has the @code{com.sun:auto-snapshot} ZFS vendor property set to
> +@code{true}.
> +
> +@item @code{auto-snapshot-keep} (default: @code{'()})
> +Specifies an association list of symbol-number pairs, indicating the number
> +of automatically-created snapshots to retain for each frequency type.
> +
> +If not specified via this field, by default there are 4 @code{frequent}, 24
> +@code{hourly}, 31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly} snapshots.
> +
> +For example:
> +
> +@lisp
> +(zfs-configuration
>
> - (kernel my-kernel)
> - (auto-snapshot-keep
> - '((frequent . 8)
> - (hourly . 12))))
>
>
>
> +@end lisp
> +
> +The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
> +@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
> +defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
> +
> +@end table
> +@end deftp
> +
> +@subsubsection ZFS Auto-Snapshot
> +
> +The ZFS service on Guix System supports auto-snapshots as implemented in the
> +Solaris operating system.
> +
> +@code{frequent} (every 15 minutes), @code{hourly}, @code{daily}, @code{weekly},
> +and @code{monthly} snapshots are created automatically for ZFS datasets that
> +have auto-snapshot enabled. They will be named, for example,
> +@code{zfs-auto-snap_frequent-2021-03-22-1415}. You can continue to use
> +manually-created snapshots as long as they do not conflict with the naming
> +convention used by auto-snapshot. You can also safely manually destroy
> +automatically-created snapshots, for example to free up space.
> +
> +The @code{com.sun:auto-snapshot} ZFS property controls auto-snapshot on a
> +per-dataset level. Sub-datasets will inherit this property from their parent
> +dataset, but can have their own property.
> +
> +You @emph{must} set this property to @code{true} or @code{false} exactly,
> +otherwise it will be treated as if the property is unset.
> +
> +For example:
> +
> +@example
> +# zfs list -o name
> +NAME
> +tank
> +tank/important-data
> +tank/tmp
> +# zfs set com.sun:auto-snapshot=true tank
> +# zfs set com.sun:auto-snapshot=false tank/tmp
> +@end example
> +
> +The above will set @code{tank} and @code{tank/important-data} to be
> +auto-snapshot, while @code{tank/tmp} will not be auto-snapshot.
> +
> +If the @code{com.sun:auto-snapshot} property is not set for a dataset
> +(the default when pools and datasets are created), then whether
> +auto-snapshot is done or not will depend on the @code{auto-snapshot?}
> +field of the @code{zfs-configuration} record.
> +
> +There are also @code{com.sun:auto-snapshot:frequent},
> +@code{com.sun:auto-snapshot:hourly}, @code{com.sun:auto-snapshot:daily},
> +@code{com.sun:auto-snapshot:weekly}, and @code{com.sun:auto-snapshot:monthly}
> +properties that give finer-grained control of whether to auto-snapshot a
> +dataset at a particular schedule.
> +
> +The number of snapshots kept for all datasets can be overridden via the
> +@code{auto-snapshot-keep} field of the @code{zfs-configuration} record.
> +There is currently no support to have different numbers of snapshots to
> +keep for different datasets.
> +
> +@subsubsection ZVOLs
> +
> +ZFS supports ZVOLs, block devices that ZFS exposes to the operating
> +system in the @code{/dev/zvol/} directory. The ZVOL will have the same
> +resilience and self-healing properties as other datasets on your ZFS pool.
> +ZVOLs can also be snapshotted (and will be included in auto-snapshotting
> +if enabled), which snapshots the state of the block device, effectively
> +snapshotting the hosted file system.
> +
> +You can put any file system inside the ZVOL. However, in order to mount this
> +file system at system start, you need to add @code{%zfs-zvol-dependency} as a
> +dependency of each file system inside a ZVOL.
> +
> +@defvr {Scheme Variable} %zfs-zvol-dependency
> +An artificial @code{<mapped-device>} which tells the file system mounting
> +service to wait for ZFS to provide ZVOLs before mounting the
> +@code{<file-system>} dependent on it.
> +@end defvr
> +
> +For example, suppose you create a ZVOL and put an ext4 filesystem
> +inside it:
> +
> +@example
> +# zfs create -V 100G tank/ext4-on-zfs
> +# mkfs.ext4 /dev/zvol/tank/ext4-on-zfs
> +# mkdir /ext4-on-zfs
> +# mount /dev/zvol/tank/ext4-on-zfs /ext4-on-zfs
> +@end example
> +
> +You can then set this up to be mounted at boot by adding this to the
> +@code{file-systems} field of your @code{operating-system} record:
> +
> +@lisp
> +(file-system
>
> - (device "/dev/zvol/tank/ext4-on-zfs")
> - (mount-point "/ext4-on-zfs")
> - (type "ext4")
> - (dependencies (list %zfs-zvol-dependency)))
> +@end lisp
>
> -
>
> +You @emph{must not} add @code{%zfs-zvol-dependency} to your
> +@code{operating-system}'s @code{mapped-devices} field, and you @emph{must
> +not} add it (or any @code{<file-system>}s dependent on it) to the
> +@code{dependencies} field of @code{zfs-configuration}. Finally, you
> +@emph{must not} use @code{%zfs-zvol-dependency} unless you actually
> +instantiate @code{zfs-service-type} on your system.
> +
> +@subsubsection Unsupported Features
> +
> +Some common features and uses of ZFS are currently not supported, or not
> +fully supported, on Guix.
> +
> +@enumerate
> +@item
> +Shepherd-managed daemons that are configured to read from or write to ZFS
> +mountpoints need to include @code{user-processes} in their @code{requirement}
> +field. This is the earliest that ZFS file systems are assured of being
> +mounted.
> +
> +Generally, most daemons will, directly or indirectly, require
> +@code{networking}, or @code{user-processes}, or both. Most implementations
> +of @code{networking} will also require @code{user-processes} so daemons that
> +require only @code{networking} will also generally start up after
> +@code{user-processes}. A notable exception, however, is
> +@code{static-networking-service-type}. You will need to explicitly add
> +@code{user-processes} as a @code{requirement} of your @code{static-networking}
> +record.
> +
> +@item
> +@code{mountpoint=legacy} ZFS file systems. The handlers for the Guix mounting
> +system have not yet been modified to support ZFS, and will expect @code{/dev}
> +paths in the @code{<file-system>}'s @code{device} field, but ZFS file systems
> +are referred to via non-path @code{pool/file/system} names. Such file systems
> +also need to be mounted @emph{after} OpenZFS has scanned for pools.
> +
> +You can still manually mount these file systems after system boot; what is
> +only unsupported is mounting them automatically at system boot by specifying
> +them in @code{<file-system>} records of your @code{operating-system}.
>
> -
>
> +@item
> +@code{/home} on ZFS. Guix will create home directories for users, but this
> +process currently cannot be scheduled after ZFS file systems are mounted.
> +Thus, the ZFS file system might be mounted @emph{after} Guix has created
> +home directories at boot, at which point OpenZFS will refuse to mount since
> +the mountpoint is not empty. However, you @emph{can} create an ext4, xfs,
> +btrfs, or other supported file system inside a ZVOL, have that depend on
> +@code{%zfs-zvol-dependency}, and set it to mount on the @code{/home}
> +directory; they will be scheduled to mount before the @code{user-homes}
> +process.
> +
> +Similarly, other locations like @code{/var}, @code{/gnu/store} and so
> +on cannot be reliably put in a ZFS file system, though they may be
> +possible to create as other file systems inside ZVOL containers.
> +
> +@item
> +@code{/} and @code{/boot} on ZFS. These require Guix to expose more of
> +the @code{initrd} very early boot process to services. It also requires
> +Guix to have the ability to explicitly load modules while still in
> +@code{initrd} (currently kernel modules loaded by
> +@code{kernel-module-loader-service-type} are loaded after @code{/} is
> +mounted). Further, since one of ZFS's main advantages is that it can
> +continue working despite the loss of one or more devices, it makes sense
> +to also support installing the bootloader on all devices of the pool that
> +contains the @code{/} and @code{/boot}; after all, if ZFS can survive the
> +loss of one device, the bootloader should also be able to survive the loss
> +of one device.
> +
> +@item
> +ZVOL swap devices. Mapped swap devices need to be listed in
> +@code{mapped-devices} to ensure they are opened before the system attempts
> +to use them, but you cannot currently add @code{%zfs-zvol-dependency} to
> +@code{mapped-devices}.
> +
> +This will also require significant amounts of testing, as various kernel
> +build options and patches may affect how swapping works, which are possibly
> +different on Guix System compared to other distributions that this feature is
> +known to work on.
> +
> +@item
> +ZFS Event Daemon. Support for this has not been written yet, patches are
> +welcome. The main issue is how to design this in a Guix style while
> +supporting legacy shell-script styles as well. In particular, OpenZFS itself
> +comes with a number of shell scripts intended for ZFS Event Daemon, and we
> +need to figure out how the user can choose to use or not use the provided
> +scripts (and configure any settings they have) or override with their own
> +custom code (which could be shell scripts they have written and trusted from
> +previous ZFS installations).
> +
> +As-is, you can create your own service that activates the ZFS Event Daemon
> +by creating the @file{/etc/zfs/zed} directory and filling it appropriately,
> +then launching @code{zed}.
> +
> +@item
> +@file{/etc/zfs/zpool.cache}. Currently the ZFS support on Guix always forces
> +scanning of all devices at bootup to look for ZFS pools. For systems with
> +dozens or hundreds of storage devices, this can lead to slow bootup. One issue
> +is that tools should really not write to @code{/etc} which is supposed to be for
> +configuration; possibly it could be moved to @code{/var} instead. Another issue
> +is that if Guix ever supports @code{/} on ZFS, we would need to somehow keep the
> +@code{zpool.cache} file inside the @code{initrd} up-to-date with what is in the
> +@code{/} mount point.
> +
> +@item
> +@code{zfs share}. This will require some (unknown amount of) work to integrate
> +into the Samba and NFS services of Guix. You @emph{can} manually set up Samba
> +and NFS to share any mounted ZFS datasets by setting up their configurations
> +properly; it just can't be done for you by @code{zfs share} and the
> +@code{sharesmb} and @code{sharenfs} properties.
> +@end enumerate
> +
> +Hopefully, support for the above only requires code to be written, and users
> +are encouraged to hack on Guix to implement the above features.
> +
> @node Mapped Devices
> @section Mapped Devices
>
> diff --git a/gnu/local.mk b/gnu/local.mk
> index d415b892e9..4147badd49 100644
> --- a/gnu/local.mk
> +++ b/gnu/local.mk
> @@ -45,6 +45,7 @@
>
> Copyright © 2021 Sharlatan Hellseher sharlatanus@gmail.com
>
> ===========================================================
>
> Copyright © 2021 Dmitry Polyakov polyakov@liltechdude.xyz
>
> ==========================================================
>
> Copyright © 2021 Andrew Tropin andrew@trop.in
>
> ==============================================
>
> +# Copyright © 2021 raid5atemyhomework raid5atemyhomework@protonmail.com
>
> ==
>
> This file is part of GNU Guix.
>
> ===============================
>
> ==
>
> @@ -633,6 +634,7 @@ GNU_SYSTEM_MODULES = \
> %D%/services/docker.scm \
> %D%/services/authentication.scm \
> %D%/services/file-sharing.scm \
>
> - %D%/services/file-systems.scm \
> %D%/services/games.scm \
> %D%/services/ganeti.scm \
> %D%/services/getmail.scm \
> diff --git a/gnu/services/base.scm b/gnu/services/base.scm
> index 50865055fe..d5d33aeada 100644
> --- a/gnu/services/base.scm
> +++ b/gnu/services/base.scm
> @@ -186,7 +186,9 @@
>
> references-file
>
>
> - %base-services))
>
>
>
> - %base-services
>
>
> -
> - dependency->shepherd-service-name))
>
>
>
> ;;; Commentary:
> ;;;
> diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
> new file mode 100644
> index 0000000000..867349c3a5
> --- /dev/null
> +++ b/gnu/services/file-systems.scm
> @@ -0,0 +1,363 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2021 raid5atemyhomework raid5atemyhomework@protonmail.com
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix. If not, see http://www.gnu.org/licenses/.
>
> -
>
> +(define-module (gnu services file-systems)
>
> - #:use-module (gnu packages file-systems)
> - #:use-module (gnu services)
> - #:use-module (gnu services base)
> - #:use-module (gnu services linux)
> - #:use-module (gnu services mcron)
> - #:use-module (gnu services shepherd)
> - #:use-module (gnu system mapped-devices)
> - #:use-module (guix gexp)
> - #:use-module (guix modules)
> - #:use-module (guix packages)
> - #:use-module (guix records)
> - #:use-module (srfi srfi-1)
> - #:export (zfs-service-type
> -
> - zfs-configuration
>
>
> - zfs-configuration?
>
>
> - zfs-configuration-kernel
>
>
> - zfs-configuration-base-zfs
>
>
> - zfs-configuration-base-zfs-auto-snapshot
>
>
> - zfs-configuration-dependencies
>
>
> - zfs-configuration-auto-mount?
>
>
> - zfs-configuration-auto-scrub
>
>
> - zfs-configuration-auto-snapshot?
>
>
> - zfs-configuration-auto-snapshot-keep
>
>
> -
> - %zfs-zvol-dependency))
>
>
> -
>
> +(define-record-type* <zfs-configuration>
>
> - zfs-configuration
>
> - make-zfs-configuration
>
> - zfs-configuration?
>
> -
> - ;; linux-libre kernel you want to compile the base-zfs module for.
>
> - (kernel zfs-configuration-kernel)
>
> -
> - ;; the OpenZFS package that will be modified to compile for the
>
> - ;; given kernel.
>
> - ;; Because it is modified and not the actual package that is used,
>
> - ;; we prepend the name 'base-'.
>
> - (base-zfs zfs-configuration-base-zfs
>
> - (default zfs))
>
>
> -
> - ;; the zfs-auto-snapshot package that will be modified to compile
>
> - ;; for the given kernel.
>
> - ;; Because it is modified and not the actual package that is used,
>
> - ;; we prepend the name 'base-'.
>
> - (base-zfs-auto-snapshot zfs-configuration-base-zfs-auto-snapshot
>
> - (default zfs-auto-snapshot))
>
>
> -
> - ;; list of <mapped-device> or <file-system> objects that must be
>
> - ;; opened/mounted before we import any ZFS pools.
>
> - (dependencies zfs-configuration-dependencies
>
> - (default '()))
>
>
> -
> - ;; #t to mount all mountable datasets by default.
>
> - ;; #f if not mounting.
>
> - ;; #t is the expected behavior on other operating systems, the
>
> - ;; #f is only supported for "rescue" operating systems where
>
> - ;; the user wants lower-level control of when to mount.
>
> - (auto-mount? zfs-configuration-auto-mount?
>
> - (default #t))
>
>
> -
> - ;; 'weekly for weekly scrubbing, 'monthly for monthly scrubbing, an
>
> - ;; mcron time specification that can be given to `job`, or #f to
>
> - ;; disable.
>
> - (auto-scrub zfs-configuration-auto-scrub
>
> - (default 'weekly))
>
>
> -
> - ;; #t to auto-snapshot by default (and `com.sun:auto-snapshot=false`
>
> - ;; disables auto-snapshot per dataset), #f to not auto-snapshot
>
> - ;; by default (and `com.sun:auto-snapshot=true` enables auto-snapshot
>
> - ;; per dataset).
>
> - (auto-snapshot? zfs-configuration-auto-snapshot?
>
> - (default #t))
>
>
> -
> - ;; association list of symbol-number pairs to indicate the number
>
> - ;; of automatic snapshots to keep for each of 'frequent, 'hourly,
>
> - ;; 'daily, 'weekly, and 'monthly.
>
> - ;; e.g. '((frequent . 8) (hourly . 12))
>
> - (auto-snapshot-keep zfs-configuration-auto-snapshot-keep
>
> - (default '())))
>
>
> -
>
> +(define %default-auto-snapshot-keep
>
> - '((frequent . 4)
> - (hourly . 24)
> - (daily . 31)
> - (weekly . 8)
> - (monthly . 12)))
> -
>
> +(define %auto-snapshot-mcron-schedule
>
> - '((frequent . "0,15,30,45 * * * *")
> - (hourly . "0 * * * *")
> - (daily . "0 0 * * *")
> - (weekly . "0 0 * * 7")
> - (monthly . "0 0 1 * *")))
> -
>
> +;; A synthetic and unusable MAPPED-DEVICE intended for use when
> +;; the user has created a mountable filesystem inside a ZFS
> +;; zvol and wants it mounted inside the configuration.scm.
> +(define %zfs-zvol-dependency
>
> - (mapped-device
> - (source '())
> - (targets '("zvol/*"))
> - (type #f)))
> -
>
> +(define (make-zfs-package conf)
>
> - "Creates a zfs package based on the given zfs-configuration.
> -
> - OpenZFS is a kernel package and to ensure best compatibility
> - it should be compiled with the specific Linux-Libre kernel
> - used on the system. This simply overrides the kernel used
> - in compilation with that given in the configuration, which
> - the user has to ensure is the same as in the operating-system."
> - (let ((kernel (zfs-configuration-kernel conf))
> - (base-zfs (zfs-configuration-base-zfs conf)))
>
>
> - (package
> - (inherit base-zfs)
>
>
> - (arguments (cons* #:linux kernel
>
>
> - (package-arguments base-zfs))))))
>
>
> -
>
> +(define (make-zfs-auto-snapshot-package conf)
>
> - "Creates a zfs-auto-snapshot package based on the given
> - zfs-configuration.
> -
> - Since the OpenZFS tools above are compiled to a specific
> - kernel version, zfs-auto-snapshot --- which calls into the
> - OpenZFS tools --- has to be compiled with the specific
> - modified OpenZFS package created in the make-zfs-package
> - procedure."
> - (let ((zfs (make-zfs-package conf))
> - (base-zfs-auto-snapshot (zfs-configuration-base-zfs-auto-snapshot conf)))
>
>
> - (package
> - (inherit base-zfs-auto-snapshot)
>
>
> - (inputs `(("zfs" ,zfs))))))
>
>
> -
>
> +(define (zfs-loadable-modules conf)
>
> - "Specifies that the specific 'module' output of the OpenZFS
> - package is to be used; for use in indicating it as a
> - loadable kernel module."
> - (list (list (make-zfs-package conf) "module")))
> -
>
> +(define (zfs-shepherd-services conf)
>
> - "Constructs a list of Shepherd services that is installed
>
> - by the ZFS Guix service.
>
> -
> - 'zfs-scan' scans all devices for ZFS pools, and makes them
>
> - available to 'zpool' commands.
>
> - 'device-mapping-zvol/' waits for /dev/zvol/ to be
>
> - populated by 'udev', and runs after 'zfs-scan'.
>
> - 'zfs-auto-mount' mounts all ZFS datasets with a 'mount'
>
> - property, which defaults to '/' followed by the name of
>
> - the dataset.
>
> -
> - All the above behavior is expected by ZFS users from
>
> - typical ZFS installations. A mild difference is that
>
> - scanning is usually based on '/etc/zfs/zpool.cache'
>
> - instead of the 'scan all devices' used below, but that
>
> - file is questionable in Guix since ideally '/etc/'
>
> - files are modified by the sysad directly;
>
> - '/etc/zfs/zpool.cache' is modified by ZFS tools."
>
> - (let* ((zfs-package (make-zfs-package conf))
>
> - (zpool (file-append zfs-package "/sbin/zpool"))
>
>
> - (zfs (file-append zfs-package "/sbin/zfs"))
>
>
> - (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
>
>
> - (scheme-modules `((srfi srfi-1)
>
>
> - (srfi srfi-34)
>
>
> - (srfi srfi-35)
>
>
> - (rnrs io ports)
>
>
> - ,@%default-modules)))
>
>
> - (define zfs-scan
>
> - (shepherd-service
>
>
> - (provision '(zfs-scan))
>
>
> - (requirement `(root-file-system
>
>
> - kernel-module-loader
>
>
> - udev
>
>
> - ,@(map dependency->shepherd-service-name
>
>
> - (zfs-configuration-dependencies conf))))
>
>
> - (documentation "Scans for and imports ZFS pools.")
>
>
> - (modules scheme-modules)
>
>
> - (start #~(lambda _
>
>
> - (guard (c ((message-condition? c)
>
>
> - (format (current-error-port)
>
>
> - "zfs: error importing pools: ~s~%"
>
>
> - (condition-message c))
>
>
> - #f))
>
>
> - ;; TODO: optionally use a cachefile.
>
>
> - (invoke #$zpool "import" "-a" "-N"))))
>
>
> - ;; Why not one-shot? Because we don't really want to rescan
>
>
> - ;; this each time a requiring process is restarted, as scanning
>
>
> - ;; can take a long time and a lot of I/O.
>
>
> - (stop #~(const #f))))
>
>
> -
> - (define device-mapping-zvol/*
>
> - (shepherd-service
>
>
> - (provision '(device-mapping-zvol/*))
>
>
> - (requirement '(zfs-scan))
>
>
> - (documentation "Waits for all ZFS ZVOLs to be opened.")
>
>
> - (modules scheme-modules)
>
>
> - (start #~(lambda _
>
>
> - (guard (c ((message-condition? c)
>
>
> - (format (current-error-port)
>
>
> - "zfs: error opening zvols: ~s~%"
>
>
> - (condition-message c))
>
>
> - #f))
>
>
> - (invoke #$zvol_wait))))
>
>
> - (stop #~(const #f))))
>
>
> -
> - (define zfs-auto-mount
>
> - (shepherd-service
>
>
> - (provision '(zfs-auto-mount))
>
>
> - (requirement '(zfs-scan))
>
>
> - (documentation "Mounts all non-legacy mounted ZFS filesystems.")
>
>
> - (modules scheme-modules)
>
>
> - (start #~(lambda _
>
>
> - (guard (c ((message-condition? c)
>
>
> - (format (current-error-port)
>
>
> - "zfs: error mounting file systems: ~s~%"
>
>
> - (condition-message c))
>
>
> - #f))
>
>
> - ;; Output to current-error-port, otherwise the
>
>
> - ;; user will not see any prompts for passwords
>
>
> - ;; of encrypted datasets.
>
>
> - ;; XXX Maybe better to explicitly open /dev/console ?
>
>
> - (with-output-to-port (current-error-port)
>
>
> - (lambda ()
>
>
> - (invoke #$zfs "mount" "-a" "-l"))))))
>
>
> - (stop #~(lambda _
>
>
> - ;; Make sure that Shepherd does not have a CWD that
>
>
> - ;; is a mounted ZFS filesystem, which would prevent
>
>
> - ;; unmounting.
>
>
> - (chdir "/")
>
>
> - (invoke #$zfs "unmount" "-a" "-f")))))
>
>
> -
> - `(,zfs-scan
>
> - ,device-mapping-zvol/*
>
>
> - ,@(if (zfs-configuration-auto-mount? conf)
>
>
> - `(,zfs-auto-mount)
>
>
> - '()))))
>
>
> -
>
> +(define (zfs-user-processes conf)
>
> - "Provides the last Shepherd service that 'user-processes' has to
> - wait for.
> -
> - If not auto-mounting, then user-processes should only wait for
> - the device scan."
> - (if (zfs-configuration-auto-mount? conf)
> - '(zfs-auto-mount)
>
>
> - '(zfs-scan)))
>
>
> -
>
> +(define (zfs-mcron-auto-snapshot-jobs conf)
>
> - "Creates a list of mcron jobs for auto-snapshotting, one for each
>
> - of the standard durations."
>
> - (let* ((user-auto-snapshot-keep (zfs-configuration-auto-snapshot-keep conf))
>
> - ;; assoc-ref has earlier entries overriding later ones.
>
>
> - (auto-snapshot-keep (append user-auto-snapshot-keep
>
>
> - %default-auto-snapshot-keep))
>
>
> - (auto-snapshot? (zfs-configuration-auto-snapshot? conf))
>
>
> - (zfs-auto-snapshot-package (make-zfs-auto-snapshot-package conf))
>
>
> - (zfs-auto-snapshot (file-append zfs-auto-snapshot-package
>
>
> - "/sbin/zfs-auto-snapshot")))
>
>
> - (map
>
> - (lambda (label)
>
>
> - (let ((keep (assoc-ref auto-snapshot-keep label))
>
>
> - (sched (assoc-ref %auto-snapshot-mcron-schedule label)))
>
>
> - #~(job '#$sched
>
>
> - (lambda ()
>
>
> - (system* #$zfs-auto-snapshot
>
>
> - "--quiet"
>
>
> - "--syslog"
>
>
> - #$(string-append "--label="
>
>
> - (symbol->string label))
>
>
> - #$(string-append "--keep="
>
>
> - (number->string keep))
>
>
> - "//")))))
>
>
> - (map first %auto-snapshot-mcron-schedule))))
>
>
> -
>
> +(define (zfs-mcron-auto-scrub-jobs conf)
>
> - "Creates a list of mcron jobs for auto-scrubbing."
> - (let* ((zfs-package (make-zfs-package conf))
> - (zpool (file-append zfs-package "/sbin/zpool"))
>
>
> - (auto-scrub (zfs-configuration-auto-scrub conf))
>
>
> - (sched (cond
>
>
> - ((eq? auto-scrub 'weekly) "0 0 * * 7")
>
>
> - ((eq? auto-scrub 'monthly) "0 0 1 * *")
>
>
> - (else auto-scrub))))
>
>
> - (define code
> - ;; We need to get access to (guix build utils) for the
>
>
> - ;; invoke procedures.
>
>
> - (with-imported-modules (source-module-closure '((guix build utils)))
>
>
> - #~(begin
>
>
> - (use-modules (guix build utils)
>
>
> - (ice-9 ports))
>
>
> - ;; The ZFS pools in the system.
>
>
> - (define pools
>
>
> - (invoke/quiet #$zpool "list" "-o" "name" "-H"))
>
>
> - ;; Only scrub if there are actual ZFS pools, as the
>
>
> - ;; zpool scrub command errors out if given an empty
>
>
> - ;; argument list.
>
>
> - (unless (null? pools)
>
>
> - ;; zpool scrub only initiates the scrub and otherwise
>
>
> - ;; prints nothing. Results are always seen on the
>
>
> - ;; zpool status command.
>
>
> - (apply invoke #$zpool "scrub" pools)))))
>
>
> - (list
> - #~(job '#$sched
>
>
> - #$(program-file "mcron-zfs-scrub.scm" code)))))
>
>
> -
>
> +(define (zfs-mcron-jobs conf)
>
> - "Creates a list of mcron jobs for ZFS management."
> - (append (zfs-mcron-auto-snapshot-jobs conf)
> - (if (zfs-configuration-auto-scrub conf)
>
>
> - (zfs-mcron-auto-scrub-jobs conf)
>
>
> - '())))
>
>
> -
>
> +(define zfs-service-type
>
> - (service-type
> - (name 'zfs)
> - (extensions
> - (list ;; Install OpenZFS kernel module into kernel profile.
>
>
> - (service-extension linux-loadable-module-service-type
>
>
> - zfs-loadable-modules)
>
>
> - ;; And load it.
>
>
> - (service-extension kernel-module-loader-service-type
>
>
> - (const '("zfs")))
>
>
> - ;; Make sure ZFS pools and datasets are mounted at
>
>
> - ;; boot.
>
>
> - (service-extension shepherd-root-service-type
>
>
> - zfs-shepherd-services)
>
>
> - ;; Make sure user-processes don't start until
>
>
> - ;; after ZFS does.
>
>
> - (service-extension user-processes-service-type
>
>
> - zfs-user-processes)
>
>
> - ;; Install automated scrubbing and snapshotting.
>
>
> - (service-extension mcron-service-type
>
>
> - zfs-mcron-jobs)
>
>
> -
> - ;; Install ZFS management commands in the system
>
>
> - ;; profile.
>
>
> - (service-extension profile-service-type
>
>
> - (compose list make-zfs-package))
>
>
> - ;; Install ZFS udev rules.
>
>
> - (service-extension udev-service-type
>
>
> - (compose list make-zfs-package))))
>
>
> - (description "Installs ZFS, an advanced filesystem and volume manager.")))
>
> base-commit: a939011b58c65f4192a10cde9e925e85702bacf4
> --
> 2.33.0
>
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type.
2021-10-19 13:18 ` raid5atemyhomework via Guix-patches via
@ 2021-10-27 7:30 ` raid5atemyhomework via Guix-patches via
2021-10-27 16:38 ` pelzflorian (Florian Pelz)
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-10-27 7:30 UTC (permalink / raw)
To: zimoun; +Cc: Maxime Devos, 45692@debbugs.gnu.org
WHEEEEEEEEEEEEEEEEEEEE ***BUMP***
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type.
2021-10-27 7:30 ` raid5atemyhomework via Guix-patches via
@ 2021-10-27 16:38 ` pelzflorian (Florian Pelz)
2021-11-30 15:26 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: pelzflorian (Florian Pelz) @ 2021-10-27 16:38 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 45692@debbugs.gnu.org, Maxime Devos, zimoun
On Wed, Oct 27, 2021 at 07:30:11AM +0000, raid5atemyhomework via Guix-patches via wrote:
> WHEEEEEEEEEEEEEEEEEEEE ***BUMP***
I’m sorry it takes so long (I’m not qualified to review), but please
note that Maxime’s e-mail setup broke down:
https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00057.html
Regards,
Florian
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type.
2021-10-27 16:38 ` pelzflorian (Florian Pelz)
@ 2021-11-30 15:26 ` raid5atemyhomework via Guix-patches via
2021-12-12 13:32 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-11-30 15:26 UTC (permalink / raw)
To: pelzflorian (Florian Pelz); +Cc: 45692@debbugs.gnu.org, Maxime Devos, zimoun
Dear Santa,
For Christmas, I would like to have a reviewer for my patch.
I have been very good this year. I have been regularly scrubbing my ZFS pool every week and I keep monthly backups of my homework now. And, I don't store my homework on broken RAID5 implementations anymore.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type.
2021-11-30 15:26 ` raid5atemyhomework via Guix-patches via
@ 2021-12-12 13:32 ` raid5atemyhomework via Guix-patches via
2021-12-21 21:15 ` [bug#45643] [PATCH 0/3] Better Support for ZFS on Guix Brice Waegeneire
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2021-12-12 13:32 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
**BUMP**
Come on, last patch, please?
If you are worried about not knowing anything about ZFS, please see this, which I used as reference: https://github.com/openzfs/zfs/discussions/11453
Otherwise, really, all I need to get reviewed is if the code is Guix-y enough to merge in.
I believe all the Scheme style guidelines are followed correctly at this point.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45643] [PATCH 0/3] Better Support for ZFS on Guix
2021-12-12 13:32 ` raid5atemyhomework via Guix-patches via
@ 2021-12-21 21:15 ` Brice Waegeneire
2022-01-01 11:59 ` [bug#45692] bug#45643: " raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: Brice Waegeneire @ 2021-12-21 21:15 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: 45643, 45692@debbugs.gnu.org
Hello raid5atemyhomework,
During the last months I had some issue swith BTRFS volumes, so I want to see if
ZFS does better in this cases.
raid5atemyhomework <raid5atemyhomework@protonmail.com> writes:
> **BUMP**
>
> Come on, last patch, please?
I'll try to review your last patch set in the comming weeks, I can't promise
anything tho but don't do ping me if I take too long.
> If you are worried about not knowing anything about ZFS, please see this,
> which I used as reference: https://github.com/openzfs/zfs/discussions/11453
Indeed that's my case, thank your for the ressource.
> Otherwise, really, all I need to get reviewed is if the code is Guix-y enough to merge in.
> I believe all the Scheme style guidelines are followed correctly at this point.
>
> Thanks
> raid5atemyhomework
Cheers,
- Brice
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] bug#45643: [PATCH 0/3] Better Support for ZFS on Guix
2021-12-21 21:15 ` [bug#45643] [PATCH 0/3] Better Support for ZFS on Guix Brice Waegeneire
@ 2022-01-01 11:59 ` raid5atemyhomework via Guix-patches via
2022-01-19 14:24 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-01-01 11:59 UTC (permalink / raw)
To: Brice Waegeneire; +Cc: 45643@debbugs.gnu.org, 45692@debbugs.gnu.org
Thanks Santa!
> Hello raid5atemyhomework,
>
> During the last months I had some issue swith BTRFS volumes, so I want to see if
> ZFS does better in this cases.
>
> raid5atemyhomework raid5atemyhomework@protonmail.com writes:
>
> > BUMP
> > Come on, last patch, please?
>
> I'll try to review your last patch set in the comming weeks, I can't promise
> anything tho but don't do ping me if I take too long.
>
> > If you are worried about not knowing anything about ZFS, please see this,
> > which I used as reference: https://github.com/openzfs/zfs/discussions/11453
>
> Indeed that's my case, thank your for the ressource.
>
> > Otherwise, really, all I need to get reviewed is if the code is Guix-y enough to merge in.
> > I believe all the Scheme style guidelines are followed correctly at this point.
> > Thanks
> > raid5atemyhomework
>
> Cheers,
>
> - Brice
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (10 preceding siblings ...)
2021-09-30 14:56 ` [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type raid5atemyhomework via Guix-patches via
@ 2022-01-07 4:21 ` raid5atemyhomework via Guix-patches via
2022-02-14 14:10 ` raid5atemyhomework via Guix-patches via
2022-03-19 14:25 ` [bug#45692] (No Subject) raid5atemyhomework via Guix-patches via
12 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-01-07 4:21 UTC (permalink / raw)
Cc: 45692@debbugs.gnu.org
Happy birthday 45692!
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] bug#45643: [PATCH 0/3] Better Support for ZFS on Guix
2022-01-01 11:59 ` [bug#45692] bug#45643: " raid5atemyhomework via Guix-patches via
@ 2022-01-19 14:24 ` raid5atemyhomework via Guix-patches via
0 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-01-19 14:24 UTC (permalink / raw)
To: Brice Waegeneire; +Cc: 45692@debbugs.gnu.org
Hi Brice,
If all you want is to get ZFS working on a Guix system: https://www.reddit.com/r/GUIX/comments/s7qu25/guide_using_zfs_on_guix/
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-01-07 4:21 ` [bug#45692] " raid5atemyhomework via Guix-patches via
@ 2022-02-14 14:10 ` raid5atemyhomework via Guix-patches via
2022-02-18 7:13 ` raid5atemyhomework via Guix-patches via
0 siblings, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-02-14 14:10 UTC (permalink / raw)
Cc: 45692@debbugs.gnu.org
BUMP
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-02-14 14:10 ` raid5atemyhomework via Guix-patches via
@ 2022-02-18 7:13 ` raid5atemyhomework via Guix-patches via
2022-03-16 23:44 ` raid5atemyhomework via Guix-patches via
2022-03-17 8:24 ` Liliana Marie Prikler
0 siblings, 2 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-02-18 7:13 UTC (permalink / raw)
Cc: 45692@debbugs.gnu.org
Modified so it applies cleanly to origin/master.
PLEASE JUST REVIEW AND MERGE, WHAT IS THE PROBLEM HERE ANYWAY?
---
From 9964668d93b496317c16c803f6d96bb3ace3560f Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Thu, 30 Sep 2021 16:58:46 +0800
Subject: [PATCH] gnu: Add ZFS service type.
* gnu/services/file-systems.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
* gnu/services/base.scm: Export dependency->shepherd-service-name.
* doc/guix.texi (ZFS File System): New subsection.
---
doc/guix.texi | 351 ++++++++++++++++++++++++++++++++
gnu/local.mk | 2 +
gnu/services/base.scm | 4 +-
gnu/services/file-systems.scm | 363 ++++++++++++++++++++++++++++++++++
4 files changed, 719 insertions(+), 1 deletion(-)
create mode 100644 gnu/services/file-systems.scm
diff --git a/doc/guix.texi b/doc/guix.texi
index c8bb484d94..65e60cd936 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -100,6 +100,7 @@ Copyright @copyright{} 2021 Josselin Poiret@*
Copyright @copyright{} 2021 Andrew Tropin@*
Copyright @copyright{} 2021 Sarah Morgensen@*
Copyright @copyright{} 2021 Josselin Poiret@*
+Copyright @copyright{} 2021 raid5atemyhomework@*
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -15594,6 +15595,356 @@ a file system declaration such as:
compress-force=zstd,space_cache=v2"))
@end lisp
+@node ZFS File System
+@subsection ZFS File System
+
+Support for ZFS file systems in Guix is based on the OpenZFS project.
+OpenZFS currently only supports Linux-Libre and is not available on the
+Hurd.
+
+OpenZFS is free software; unfortunately its license is incompatible with
+the GNU General Public License (GPL), the license of the Linux kernel,
+which means they cannot be distributed together. However, as a user,
+you can choose to build ZFS and use it together with Linux; you can
+even rely on Guix to automate this task. See
+@uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
+the Free Software Foundation} for more information.
+
+As a large and complex kernel module, OpenZFS has to be compiled for a
+specific version of Linux-Libre. At times, the latest OpenZFS package
+available in Guix is not compatible with the latest Linux-Libre version.
+Thus, directly installing the @code{zfs} package can fail.
+
+Instead, you are recommended to select a specific older long-term-support
+Linux-Libre kernel. Do not use @code{linux-libre-lts}, as even the
+latest long-term-support kernel may be too new for @code{zfs}. Instead,
+explicitly select a specific older version, such as @code{linux-libre-5.10},
+and upgrade it manually later as new long-term-support kernels become
+available that you have confirmed is compatible with the latest available
+OpenZFS version on Guix.
+
+For example, you can modify your system configuration file to a specific
+Linux-Libre version and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+ #;@dots{}
+ linux)
+(use-service-modules
+ #;@dots{}
+ file-systems)
+
+(define my-kernel linux-libre-5.10)
+
+(operating-system
+ (kernel my-kernel)
+ #;@dots{}
+ (services
+ (cons* (service zfs-service-type
+ (zfs-configuration
+ (kernel my-kernel)))
+ #;@dots{}
+ %desktop-services))
+ #;@dots{})
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type for a service that adds ZFS support to your operating
+system. The service is configured using a @code{zfs-configuration}
+record.
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+ (zfs-configuration
+ (kernel linux-libre-5.4)))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS support in Guix
+System. Its fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux-Libre kernel to compile OpenZFS for. This field
+is always required. It @emph{must} be the same kernel you use in your
+@code{operating-system} form.
+
+@item @code{base-zfs} (default: @code{zfs})
+The OpenZFS package that will be compiled for the given Linux-Libre kernel.
+
+@item @code{base-zfs-auto-snapshot} (default: @code{zfs-auto-snapshot})
+The @code{zfs-auto-snapshot} package to use. It will be modified to
+specifically use the OpenZFS compiled for your kernel.
+
+@item @code{dependencies} (default: @code{'()})
+A list of @code{<file-system>} or @code{<mapped-device>} records that must
+be mounted or opened before OpenZFS scans for pools to import. For example,
+if you have set up LUKS containers as leaf VDEVs in a pool, you have to
+include their corresponding @code{<mapped-ddevice>} records so that OpenZFS
+can import the pool correctly at bootup.
+
+@item @code{auto-mount?} (default: @code{#t})
+Whether to mount datasets with the ZFS @code{mountpoint} property automatically
+at startup. This is the behavior that ZFS users usually expect. You might
+set this to @code{#f} for an operating system intended as a ``rescue'' system
+that is intended to help debug problems with the disks rather than actually
+work in production.
+
+@item @code{auto-scrub} (default: @code{'weekly})
+Specifies how often to scrub all pools. Can be the symbols @code{'weekly} or
+@code{'monthly}, or a schedule specification understood by
+@xref{mcron, mcron job specifications,, mcron, GNU@tie{}mcron}, such as
+@code{"0 3 * * 6"} for ``every 3AM on Saturday''.
+It can also be @code{#f} to disable auto-scrubbing (@strong{not recommended}).
+
+The general guideline is to scrub weekly when using consumer-quality drives, and
+to scrub monthly when using enterprise-quality drives.
+
+@code{'weekly} scrubs are done on Sunday midnight, while @code{monthly} scrubs
+are done on midnight on the first day of each month.
+
+@item @code{auto-snapshot?} (default: @code{#t})
+Specifies whether to auto-snapshot by default. If @code{#t}, then snapshots
+are automatically created except for ZFS datasets with the
+@code{com.sun:auto-snapshot} ZFS vendor property set to @code{false}.
+
+If @code{#f}, snapshots will not be automatically created, unless the ZFS
+dataset has the @code{com.sun:auto-snapshot} ZFS vendor property set to
+@code{true}.
+
+@item @code{auto-snapshot-keep} (default: @code{'()})
+Specifies an association list of symbol-number pairs, indicating the number
+of automatically-created snapshots to retain for each frequency type.
+
+If not specified via this field, by default there are 4 @code{frequent}, 24
+@code{hourly}, 31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly} snapshots.
+
+For example:
+
+@lisp
+(zfs-configuration
+ (kernel my-kernel)
+ (auto-snapshot-keep
+ '((frequent . 8)
+ (hourly . 12))))
+@end lisp
+
+The above will keep 8 @code{frequent} snapshots and 12 @code{hourly} snapshots.
+@code{daily}, @code{weekly}, and @code{monthly} snapshots will keep their
+defaults (31 @code{daily}, 8 @code{weekly}, and 12 @code{monthly}).
+
+@end table
+@end deftp
+
+@subsubsection ZFS Auto-Snapshot
+
+The ZFS service on Guix System supports auto-snapshots as implemented in the
+Solaris operating system.
+
+@code{frequent} (every 15 minutes), @code{hourly}, @code{daily}, @code{weekly},
+and @code{monthly} snapshots are created automatically for ZFS datasets that
+have auto-snapshot enabled. They will be named, for example,
+@code{zfs-auto-snap_frequent-2021-03-22-1415}. You can continue to use
+manually-created snapshots as long as they do not conflict with the naming
+convention used by auto-snapshot. You can also safely manually destroy
+automatically-created snapshots, for example to free up space.
+
+The @code{com.sun:auto-snapshot} ZFS property controls auto-snapshot on a
+per-dataset level. Sub-datasets will inherit this property from their parent
+dataset, but can have their own property.
+
+You @emph{must} set this property to @code{true} or @code{false} exactly,
+otherwise it will be treated as if the property is unset.
+
+For example:
+
+@example
+# zfs list -o name
+NAME
+tank
+tank/important-data
+tank/tmp
+# zfs set com.sun:auto-snapshot=true tank
+# zfs set com.sun:auto-snapshot=false tank/tmp
+@end example
+
+The above will set @code{tank} and @code{tank/important-data} to be
+auto-snapshot, while @code{tank/tmp} will not be auto-snapshot.
+
+If the @code{com.sun:auto-snapshot} property is not set for a dataset
+(the default when pools and datasets are created), then whether
+auto-snapshot is done or not will depend on the @code{auto-snapshot?}
+field of the @code{zfs-configuration} record.
+
+There are also @code{com.sun:auto-snapshot:frequent},
+@code{com.sun:auto-snapshot:hourly}, @code{com.sun:auto-snapshot:daily},
+@code{com.sun:auto-snapshot:weekly}, and @code{com.sun:auto-snapshot:monthly}
+properties that give finer-grained control of whether to auto-snapshot a
+dataset at a particular schedule.
+
+The number of snapshots kept for all datasets can be overridden via the
+@code{auto-snapshot-keep} field of the @code{zfs-configuration} record.
+There is currently no support to have different numbers of snapshots to
+keep for different datasets.
+
+@subsubsection ZVOLs
+
+ZFS supports ZVOLs, block devices that ZFS exposes to the operating
+system in the @code{/dev/zvol/} directory. The ZVOL will have the same
+resilience and self-healing properties as other datasets on your ZFS pool.
+ZVOLs can also be snapshotted (and will be included in auto-snapshotting
+if enabled), which snapshots the state of the block device, effectively
+snapshotting the hosted file system.
+
+You can put any file system inside the ZVOL. However, in order to mount this
+file system at system start, you need to add @code{%zfs-zvol-dependency} as a
+dependency of each file system inside a ZVOL.
+
+@defvr {Scheme Variable} %zfs-zvol-dependency
+An artificial @code{<mapped-device>} which tells the file system mounting
+service to wait for ZFS to provide ZVOLs before mounting the
+@code{<file-system>} dependent on it.
+@end defvr
+
+For example, suppose you create a ZVOL and put an ext4 filesystem
+inside it:
+
+@example
+# zfs create -V 100G tank/ext4-on-zfs
+# mkfs.ext4 /dev/zvol/tank/ext4-on-zfs
+# mkdir /ext4-on-zfs
+# mount /dev/zvol/tank/ext4-on-zfs /ext4-on-zfs
+@end example
+
+You can then set this up to be mounted at boot by adding this to the
+@code{file-systems} field of your @code{operating-system} record:
+
+@lisp
+(file-system
+ (device "/dev/zvol/tank/ext4-on-zfs")
+ (mount-point "/ext4-on-zfs")
+ (type "ext4")
+ (dependencies (list %zfs-zvol-dependency)))
+@end lisp
+
+You @emph{must not} add @code{%zfs-zvol-dependency} to your
+@code{operating-system}'s @code{mapped-devices} field, and you @emph{must
+not} add it (or any @code{<file-system>}s dependent on it) to the
+@code{dependencies} field of @code{zfs-configuration}. Finally, you
+@emph{must not} use @code{%zfs-zvol-dependency} unless you actually
+instantiate @code{zfs-service-type} on your system.
+
+@subsubsection Unsupported Features
+
+Some common features and uses of ZFS are currently not supported, or not
+fully supported, on Guix.
+
+@enumerate
+@item
+Shepherd-managed daemons that are configured to read from or write to ZFS
+mountpoints need to include @code{user-processes} in their @code{requirement}
+field. This is the earliest that ZFS file systems are assured of being
+mounted.
+
+Generally, most daemons will, directly or indirectly, require
+@code{networking}, or @code{user-processes}, or both. Most implementations
+of @code{networking} will also require @code{user-processes} so daemons that
+require only @code{networking} will also generally start up after
+@code{user-processes}. A notable exception, however, is
+@code{static-networking-service-type}. You will need to explicitly add
+@code{user-processes} as a @code{requirement} of your @code{static-networking}
+record.
+
+@item
+@code{mountpoint=legacy} ZFS file systems. The handlers for the Guix mounting
+system have not yet been modified to support ZFS, and will expect @code{/dev}
+paths in the @code{<file-system>}'s @code{device} field, but ZFS file systems
+are referred to via non-path @code{pool/file/system} names. Such file systems
+also need to be mounted @emph{after} OpenZFS has scanned for pools.
+
+You can still manually mount these file systems after system boot; what is
+only unsupported is mounting them automatically at system boot by specifying
+them in @code{<file-system>} records of your @code{operating-system}.
+
+@item
+@code{/home} on ZFS. Guix will create home directories for users, but this
+process currently cannot be scheduled after ZFS file systems are mounted.
+Thus, the ZFS file system might be mounted @emph{after} Guix has created
+home directories at boot, at which point OpenZFS will refuse to mount since
+the mountpoint is not empty. However, you @emph{can} create an ext4, xfs,
+btrfs, or other supported file system inside a ZVOL, have that depend on
+@code{%zfs-zvol-dependency}, and set it to mount on the @code{/home}
+directory; they will be scheduled to mount before the @code{user-homes}
+process.
+
+Similarly, other locations like @code{/var}, @code{/gnu/store} and so
+on cannot be reliably put in a ZFS file system, though they may be
+possible to create as other file systems inside ZVOL containers.
+
+@item
+@code{/} and @code{/boot} on ZFS. These require Guix to expose more of
+the @code{initrd} very early boot process to services. It also requires
+Guix to have the ability to explicitly load modules while still in
+@code{initrd} (currently kernel modules loaded by
+@code{kernel-module-loader-service-type} are loaded after @code{/} is
+mounted). Further, since one of ZFS's main advantages is that it can
+continue working despite the loss of one or more devices, it makes sense
+to also support installing the bootloader on all devices of the pool that
+contains the @code{/} and @code{/boot}; after all, if ZFS can survive the
+loss of one device, the bootloader should also be able to survive the loss
+of one device.
+
+@item
+ZVOL swap devices. Mapped swap devices need to be listed in
+@code{mapped-devices} to ensure they are opened before the system attempts
+to use them, but you cannot currently add @code{%zfs-zvol-dependency} to
+@code{mapped-devices}.
+
+This will also require significant amounts of testing, as various kernel
+build options and patches may affect how swapping works, which are possibly
+different on Guix System compared to other distributions that this feature is
+known to work on.
+
+@item
+ZFS Event Daemon. Support for this has not been written yet, patches are
+welcome. The main issue is how to design this in a Guix style while
+supporting legacy shell-script styles as well. In particular, OpenZFS itself
+comes with a number of shell scripts intended for ZFS Event Daemon, and we
+need to figure out how the user can choose to use or not use the provided
+scripts (and configure any settings they have) or override with their own
+custom code (which could be shell scripts they have written and trusted from
+previous ZFS installations).
+
+As-is, you can create your own service that activates the ZFS Event Daemon
+by creating the @file{/etc/zfs/zed} directory and filling it appropriately,
+then launching @code{zed}.
+
+@item
+@file{/etc/zfs/zpool.cache}. Currently the ZFS support on Guix always forces
+scanning of all devices at bootup to look for ZFS pools. For systems with
+dozens or hundreds of storage devices, this can lead to slow bootup. One issue
+is that tools should really not write to @code{/etc} which is supposed to be for
+configuration; possibly it could be moved to @code{/var} instead. Another issue
+is that if Guix ever supports @code{/} on ZFS, we would need to somehow keep the
+@code{zpool.cache} file inside the @code{initrd} up-to-date with what is in the
+@code{/} mount point.
+
+@item
+@code{zfs share}. This will require some (unknown amount of) work to integrate
+into the Samba and NFS services of Guix. You @emph{can} manually set up Samba
+and NFS to share any mounted ZFS datasets by setting up their configurations
+properly; it just can't be done for you by @code{zfs share} and the
+@code{sharesmb} and @code{sharenfs} properties.
+@end enumerate
+
+Hopefully, support for the above only requires code to be written, and users
+are encouraged to hack on Guix to implement the above features.
+
+
@node Mapped Devices
@section Mapped Devices
diff --git a/gnu/local.mk b/gnu/local.mk
index 1252643dc0..95e31a4b7e 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -49,6 +49,7 @@
# Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
# Copyright © 2022 Daniel Meißner <daniel.meissner-i4k@ruhr-uni-bochum.de>
# Copyright © 2022 Remco van 't Veer <remco@remworks.net>
+# Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
#
# This file is part of GNU Guix.
#
@@ -648,6 +649,7 @@ GNU_SYSTEM_MODULES = \
%D%/services/docker.scm \
%D%/services/authentication.scm \
%D%/services/file-sharing.scm \
+ %D%/services/file-systems.scm \
%D%/services/games.scm \
%D%/services/ganeti.scm \
%D%/services/getmail.scm \
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index fbd01e84d6..aacb9e5e1b 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -220,7 +220,9 @@ (define-module (gnu services base)
references-file
- %base-services))
+ %base-services
+
+ dependency->shepherd-service-name))
;;; Commentary:
;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..867349c3a5
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,363 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+ #:use-module (gnu packages file-systems)
+ #:use-module (gnu services)
+ #:use-module (gnu services base)
+ #:use-module (gnu services linux)
+ #:use-module (gnu services mcron)
+ #:use-module (gnu services shepherd)
+ #:use-module (gnu system mapped-devices)
+ #:use-module (guix gexp)
+ #:use-module (guix modules)
+ #:use-module (guix packages)
+ #:use-module (guix records)
+ #:use-module (srfi srfi-1)
+ #:export (zfs-service-type
+
+ zfs-configuration
+ zfs-configuration?
+ zfs-configuration-kernel
+ zfs-configuration-base-zfs
+ zfs-configuration-base-zfs-auto-snapshot
+ zfs-configuration-dependencies
+ zfs-configuration-auto-mount?
+ zfs-configuration-auto-scrub
+ zfs-configuration-auto-snapshot?
+ zfs-configuration-auto-snapshot-keep
+
+ %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+ zfs-configuration
+ make-zfs-configuration
+ zfs-configuration?
+
+ ;; linux-libre kernel you want to compile the base-zfs module for.
+ (kernel zfs-configuration-kernel)
+
+ ;; the OpenZFS package that will be modified to compile for the
+ ;; given kernel.
+ ;; Because it is modified and not the actual package that is used,
+ ;; we prepend the name 'base-'.
+ (base-zfs zfs-configuration-base-zfs
+ (default zfs))
+
+ ;; the zfs-auto-snapshot package that will be modified to compile
+ ;; for the given kernel.
+ ;; Because it is modified and not the actual package that is used,
+ ;; we prepend the name 'base-'.
+ (base-zfs-auto-snapshot zfs-configuration-base-zfs-auto-snapshot
+ (default zfs-auto-snapshot))
+
+ ;; list of <mapped-device> or <file-system> objects that must be
+ ;; opened/mounted before we import any ZFS pools.
+ (dependencies zfs-configuration-dependencies
+ (default '()))
+
+ ;; #t to mount all mountable datasets by default.
+ ;; #f if not mounting.
+ ;; #t is the expected behavior on other operating systems, the
+ ;; #f is only supported for "rescue" operating systems where
+ ;; the user wants lower-level control of when to mount.
+ (auto-mount? zfs-configuration-auto-mount?
+ (default #t))
+
+ ;; 'weekly for weekly scrubbing, 'monthly for monthly scrubbing, an
+ ;; mcron time specification that can be given to `job`, or #f to
+ ;; disable.
+ (auto-scrub zfs-configuration-auto-scrub
+ (default 'weekly))
+
+ ;; #t to auto-snapshot by default (and `com.sun:auto-snapshot=false`
+ ;; disables auto-snapshot per dataset), #f to not auto-snapshot
+ ;; by default (and `com.sun:auto-snapshot=true` enables auto-snapshot
+ ;; per dataset).
+ (auto-snapshot? zfs-configuration-auto-snapshot?
+ (default #t))
+
+ ;; association list of symbol-number pairs to indicate the number
+ ;; of automatic snapshots to keep for each of 'frequent, 'hourly,
+ ;; 'daily, 'weekly, and 'monthly.
+ ;; e.g. '((frequent . 8) (hourly . 12))
+ (auto-snapshot-keep zfs-configuration-auto-snapshot-keep
+ (default '())))
+
+(define %default-auto-snapshot-keep
+ '((frequent . 4)
+ (hourly . 24)
+ (daily . 31)
+ (weekly . 8)
+ (monthly . 12)))
+
+(define %auto-snapshot-mcron-schedule
+ '((frequent . "0,15,30,45 * * * *")
+ (hourly . "0 * * * *")
+ (daily . "0 0 * * *")
+ (weekly . "0 0 * * 7")
+ (monthly . "0 0 1 * *")))
+
+;; A synthetic and unusable MAPPED-DEVICE intended for use when
+;; the user has created a mountable filesystem inside a ZFS
+;; zvol and wants it mounted inside the configuration.scm.
+(define %zfs-zvol-dependency
+ (mapped-device
+ (source '())
+ (targets '("zvol/*"))
+ (type #f)))
+
+(define (make-zfs-package conf)
+ "Creates a zfs package based on the given zfs-configuration.
+
+ OpenZFS is a kernel package and to ensure best compatibility
+ it should be compiled with the specific Linux-Libre kernel
+ used on the system. This simply overrides the kernel used
+ in compilation with that given in the configuration, which
+ the user has to ensure is the same as in the operating-system."
+ (let ((kernel (zfs-configuration-kernel conf))
+ (base-zfs (zfs-configuration-base-zfs conf)))
+ (package
+ (inherit base-zfs)
+ (arguments (cons* #:linux kernel
+ (package-arguments base-zfs))))))
+
+(define (make-zfs-auto-snapshot-package conf)
+ "Creates a zfs-auto-snapshot package based on the given
+ zfs-configuration.
+
+ Since the OpenZFS tools above are compiled to a specific
+ kernel version, zfs-auto-snapshot --- which calls into the
+ OpenZFS tools --- has to be compiled with the specific
+ modified OpenZFS package created in the make-zfs-package
+ procedure."
+ (let ((zfs (make-zfs-package conf))
+ (base-zfs-auto-snapshot (zfs-configuration-base-zfs-auto-snapshot conf)))
+ (package
+ (inherit base-zfs-auto-snapshot)
+ (inputs `(("zfs" ,zfs))))))
+
+(define (zfs-loadable-modules conf)
+ "Specifies that the specific 'module' output of the OpenZFS
+ package is to be used; for use in indicating it as a
+ loadable kernel module."
+ (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+ "Constructs a list of Shepherd services that is installed
+ by the ZFS Guix service.
+
+ 'zfs-scan' scans all devices for ZFS pools, and makes them
+ available to 'zpool' commands.
+ 'device-mapping-zvol/*' waits for /dev/zvol/* to be
+ populated by 'udev', and runs after 'zfs-scan'.
+ 'zfs-auto-mount' mounts all ZFS datasets with a 'mount'
+ property, which defaults to '/' followed by the name of
+ the dataset.
+
+ All the above behavior is expected by ZFS users from
+ typical ZFS installations. A mild difference is that
+ scanning is usually based on '/etc/zfs/zpool.cache'
+ instead of the 'scan all devices' used below, but that
+ file is questionable in Guix since ideally '/etc/'
+ files are modified by the sysad directly;
+ '/etc/zfs/zpool.cache' is modified by ZFS tools."
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (zfs (file-append zfs-package "/sbin/zfs"))
+ (zvol_wait (file-append zfs-package "/bin/zvol_wait"))
+ (scheme-modules `((srfi srfi-1)
+ (srfi srfi-34)
+ (srfi srfi-35)
+ (rnrs io ports)
+ ,@%default-modules)))
+ (define zfs-scan
+ (shepherd-service
+ (provision '(zfs-scan))
+ (requirement `(root-file-system
+ kernel-module-loader
+ udev
+ ,@(map dependency->shepherd-service-name
+ (zfs-configuration-dependencies conf))))
+ (documentation "Scans for and imports ZFS pools.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error importing pools: ~s~%"
+ (condition-message c))
+ #f))
+ ;; TODO: optionally use a cachefile.
+ (invoke #$zpool "import" "-a" "-N"))))
+ ;; Why not one-shot? Because we don't really want to rescan
+ ;; this each time a requiring process is restarted, as scanning
+ ;; can take a long time and a lot of I/O.
+ (stop #~(const #f))))
+
+ (define device-mapping-zvol/*
+ (shepherd-service
+ (provision '(device-mapping-zvol/*))
+ (requirement '(zfs-scan))
+ (documentation "Waits for all ZFS ZVOLs to be opened.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error opening zvols: ~s~%"
+ (condition-message c))
+ #f))
+ (invoke #$zvol_wait))))
+ (stop #~(const #f))))
+
+ (define zfs-auto-mount
+ (shepherd-service
+ (provision '(zfs-auto-mount))
+ (requirement '(zfs-scan))
+ (documentation "Mounts all non-legacy mounted ZFS filesystems.")
+ (modules scheme-modules)
+ (start #~(lambda _
+ (guard (c ((message-condition? c)
+ (format (current-error-port)
+ "zfs: error mounting file systems: ~s~%"
+ (condition-message c))
+ #f))
+ ;; Output to current-error-port, otherwise the
+ ;; user will not see any prompts for passwords
+ ;; of encrypted datasets.
+ ;; XXX Maybe better to explicitly open /dev/console ?
+ (with-output-to-port (current-error-port)
+ (lambda ()
+ (invoke #$zfs "mount" "-a" "-l"))))))
+ (stop #~(lambda _
+ ;; Make sure that Shepherd does not have a CWD that
+ ;; is a mounted ZFS filesystem, which would prevent
+ ;; unmounting.
+ (chdir "/")
+ (invoke #$zfs "unmount" "-a" "-f")))))
+
+ `(,zfs-scan
+ ,device-mapping-zvol/*
+ ,@(if (zfs-configuration-auto-mount? conf)
+ `(,zfs-auto-mount)
+ '()))))
+
+(define (zfs-user-processes conf)
+ "Provides the last Shepherd service that 'user-processes' has to
+ wait for.
+
+ If not auto-mounting, then user-processes should only wait for
+ the device scan."
+ (if (zfs-configuration-auto-mount? conf)
+ '(zfs-auto-mount)
+ '(zfs-scan)))
+
+(define (zfs-mcron-auto-snapshot-jobs conf)
+ "Creates a list of mcron jobs for auto-snapshotting, one for each
+ of the standard durations."
+ (let* ((user-auto-snapshot-keep (zfs-configuration-auto-snapshot-keep conf))
+ ;; assoc-ref has earlier entries overriding later ones.
+ (auto-snapshot-keep (append user-auto-snapshot-keep
+ %default-auto-snapshot-keep))
+ (auto-snapshot? (zfs-configuration-auto-snapshot? conf))
+ (zfs-auto-snapshot-package (make-zfs-auto-snapshot-package conf))
+ (zfs-auto-snapshot (file-append zfs-auto-snapshot-package
+ "/sbin/zfs-auto-snapshot")))
+ (map
+ (lambda (label)
+ (let ((keep (assoc-ref auto-snapshot-keep label))
+ (sched (assoc-ref %auto-snapshot-mcron-schedule label)))
+ #~(job '#$sched
+ (lambda ()
+ (system* #$zfs-auto-snapshot
+ "--quiet"
+ "--syslog"
+ #$(string-append "--label="
+ (symbol->string label))
+ #$(string-append "--keep="
+ (number->string keep))
+ "//")))))
+ (map first %auto-snapshot-mcron-schedule))))
+
+(define (zfs-mcron-auto-scrub-jobs conf)
+ "Creates a list of mcron jobs for auto-scrubbing."
+ (let* ((zfs-package (make-zfs-package conf))
+ (zpool (file-append zfs-package "/sbin/zpool"))
+ (auto-scrub (zfs-configuration-auto-scrub conf))
+ (sched (cond
+ ((eq? auto-scrub 'weekly) "0 0 * * 7")
+ ((eq? auto-scrub 'monthly) "0 0 1 * *")
+ (else auto-scrub))))
+ (define code
+ ;; We need to get access to (guix build utils) for the
+ ;; invoke procedures.
+ (with-imported-modules (source-module-closure '((guix build utils)))
+ #~(begin
+ (use-modules (guix build utils)
+ (ice-9 ports))
+ ;; The ZFS pools in the system.
+ (define pools
+ (invoke/quiet #$zpool "list" "-o" "name" "-H"))
+ ;; Only scrub if there are actual ZFS pools, as the
+ ;; zpool scrub command errors out if given an empty
+ ;; argument list.
+ (unless (null? pools)
+ ;; zpool scrub only initiates the scrub and otherwise
+ ;; prints nothing. Results are always seen on the
+ ;; zpool status command.
+ (apply invoke #$zpool "scrub" pools)))))
+ (list
+ #~(job '#$sched
+ #$(program-file "mcron-zfs-scrub.scm" code)))))
+
+(define (zfs-mcron-jobs conf)
+ "Creates a list of mcron jobs for ZFS management."
+ (append (zfs-mcron-auto-snapshot-jobs conf)
+ (if (zfs-configuration-auto-scrub conf)
+ (zfs-mcron-auto-scrub-jobs conf)
+ '())))
+
+(define zfs-service-type
+ (service-type
+ (name 'zfs)
+ (extensions
+ (list ;; Install OpenZFS kernel module into kernel profile.
+ (service-extension linux-loadable-module-service-type
+ zfs-loadable-modules)
+ ;; And load it.
+ (service-extension kernel-module-loader-service-type
+ (const '("zfs")))
+ ;; Make sure ZFS pools and datasets are mounted at
+ ;; boot.
+ (service-extension shepherd-root-service-type
+ zfs-shepherd-services)
+ ;; Make sure user-processes don't start until
+ ;; after ZFS does.
+ (service-extension user-processes-service-type
+ zfs-user-processes)
+ ;; Install automated scrubbing and snapshotting.
+ (service-extension mcron-service-type
+ zfs-mcron-jobs)
+
+ ;; Install ZFS management commands in the system
+ ;; profile.
+ (service-extension profile-service-type
+ (compose list make-zfs-package))
+ ;; Install ZFS udev rules.
+ (service-extension udev-service-type
+ (compose list make-zfs-package))))
+ (description "Installs ZFS, an advanced filesystem and volume manager.")))
base-commit: 1d1a4efd8cdac3757792cfbae92440edc5c3a802
--
2.34.0
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-02-18 7:13 ` raid5atemyhomework via Guix-patches via
@ 2022-03-16 23:44 ` raid5atemyhomework via Guix-patches via
2022-03-17 8:24 ` Liliana Marie Prikler
1 sibling, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-03-16 23:44 UTC (permalink / raw)
Cc: 45692@debbugs.gnu.org
BUMP
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-02-18 7:13 ` raid5atemyhomework via Guix-patches via
2022-03-16 23:44 ` raid5atemyhomework via Guix-patches via
@ 2022-03-17 8:24 ` Liliana Marie Prikler
2022-03-17 17:22 ` Maxime Devos
2022-03-19 14:09 ` raid5atemyhomework via Guix-patches via
1 sibling, 2 replies; 81+ messages in thread
From: Liliana Marie Prikler @ 2022-03-17 8:24 UTC (permalink / raw)
To: raid5atemyhomework; +Cc: Maxime Devos, 45692@debbugs.gnu.org
Hi raid5,
Am Freitag, dem 18.02.2022 um 07:13 +0000 schrieb raid5atemyhomework:
> Modified so it applies cleanly to origin/master.
>
> PLEASE JUST REVIEW AND MERGE, WHAT IS THE PROBLEM HERE ANYWAY?
You've been begging for review for a while now, so let me inform you
that the way you've been doing this is not particularly helpful to you
or the reviewers.
First of all, your follow-up messages do not include anyone who has so
far reviewed the patch in the "To:" or "Cc:" field. This makes it less
likely that they will actually see your message. Secondly, the tone in
which you're asking is not nice to the reviewers. I can understand
you're a little frustrated waiting for so long, but shouting "WHAT IS
THE PROBLEM ANYWAY?" communicates that you're both unaware of and do
not care about burdens (e.g. maintenance) that are created by your
patch. This in turn prompts reviewers to look away; both out of spite
and in order not to deal with this mess at all.
I have no stake in ZFS and no intent to review this patch beyond this
point, but here a few questions to ask: Why is it necessary to define a
file system as a service? Why do we need to export a seemingly
unrelated variable? Can this be tested? Is this sufficiently tested?
Are there any points Maxime that were drowned out by a huge wall of
licensing-related messages being passed back and forth that I will not
attempt to sift through in order to respond to this message? If so,
have those been sufficiently addressed?
Another complicating factor for this bug in particular is that the mumi
web interface and the raw messages are out of sync; I have no idea why
that is the case, but trying to fetch a patch only to get one of your
bump messages is not particularly encouraging.
In any case, I've added Maxime to CC so they can have a closer look at
it.
Cheers
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-03-17 8:24 ` Liliana Marie Prikler
@ 2022-03-17 17:22 ` Maxime Devos
2022-03-17 18:38 ` zimoun
2022-03-19 14:24 ` raid5atemyhomework via Guix-patches via
2022-03-19 14:09 ` raid5atemyhomework via Guix-patches via
1 sibling, 2 replies; 81+ messages in thread
From: Maxime Devos @ 2022-03-17 17:22 UTC (permalink / raw)
To: Liliana Marie Prikler, raid5atemyhomework; +Cc: 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 5408 bytes --]
Liliana Marie Prikler schreef op do 17-03-2022 om 09:24 [+0100]:
> In any case, I've added Maxime to CC so they can have a closer look at
> it.
I have been more-or-less ignoring the ZFS patches since some time after
<https://issues.guix.gnu.org/45692#44>. If ZFS people(^), after a
disagreement about licensing concerns, directly jump to accusations of
gaslighting and sabogate, completely ignoring my previous arguments (*)
without trying to refute any of them or bringing new arguments, then I
don't want to be involved with ZFS.
(^) So far only Mason Loring Bliss, _not_ raid5atemyhomework!
Also, the various ‘work-arounds’ around the GPL<->CDLL incompatibility
still seem super fishy to me even if they _might_ be technically
correct. To me, this makes reviewing the code practically pointless --
why review the zfs service patches if they will have to be reverted
due to incompatibility concerns anyway? Summarised:
* The ‘Oracle does not care so no legal risk’ argument:
- Oracle might not care, but there are other parties involved as
well (e.g. the Linux people and contributors to OpenZFS).
- Not getting caught doesn't mean things are above board. It just
means you haven't got caught, and you might get caught later.
- Has anyone actually ever asked Oracle for some official ‘yes,
go ahead’ / ‘no, here's a DMCA notice / see you at YYYY-MM-DD
in court’ / ‘no, but you're too small fry to bother so you'll
get away for it ... for now’ response?
AFAICT, this has not been done.
- Even if it would be very strange for Oracle to try to stop (the
Linux part of(*)) OpenZFS, why would that stop Oracle and how
would the odd behaviour actually legally matter?
* The ‘zfs package is already in Guix’ argument
(https://issues.guix.gnu.org/45692#47): then it should be
reverted when the incompatibility is discovered.
Also, the incompatibility issue has been noted before:
https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00404.html
though it appears to have been forgotten in
https://lists.gnu.org/archive/html/guix-patches/2019-12/msg00543.html
presumably because different people were involved?
* The ‘Guix is not distributing the source code, it's only pointing
to the source code’ argument:
- We do distribute the source code, at https://ci.guix.gnu.org
- probably also via our friends at SWH
- and via the Wayback Machine fallback
- possibly also to any Guix users on the local network, when using
'--advertise' and '--discover'
- and by delegating the distributing to the OpenZFS project
- even if pointing to the tarball OpenZFS web would not count as
distribution, assuming there's a license incompatibility (and
hence, the Linux part of OpenZFS is illegal (*), wouldn't this
pointing count as facilitation of a crime (or misdemeanor or
contract breach or whatever's the local terminology), wouldn't
this make Guix or individuals behind Guix accomplishes?
- even if it's all legal, what about freedom 3 -- the freedom
to distribute the program?
- also, not being able to distribute the source code by ourselves
seems rather inconvenient
* The ‘we're not doing binary distribution’ argument:
- That seems rather inconvenient, why not use BTRFS instead which
seems quite capable and doesn't have this weird restriction?
- Freedom 3 is:
‘The freedom to redistribute copies so you can help others
(freedom 2).’ Guix redistributes copies is a convenient form,
to help all users (‘others’). To help the users, it not only
redistributes in source form, but also in binary form
(substitutes). But the CDDL+GPL combination stops us from
helping others by redistributing binary copies!
Basically, if there's the freedom to redistribute copies,
shouldn't this include _binary_ copies, especially when
binaries are convenient?
- We _are_ doing binary distribution:
(here, ‘we’ includes all relevant users of Guix)
An uninitiated user might do "guix system image ..." to
produce an image (that happens to include a binary ZFS),
dutifully uses "guix build --sources=transitive" and share
the sources+binary with other people, and accidentally commit
a violation.
All the initrd, system image and "guix pack" would need to
propagate unsubstitutability (and the top-level tools might need
to error out) and this needs to be tested, AFAIK this has not
been done.
* The ‘we're not distributing _modified_ source code’ argument:
Freedom 4! We should be able to (legally) distribute modified
source code as well.
* Various ’technically, because of Section 1 (bis) alpha Z of this
license, Paragraph 2 beta 3 of that license, this and that clause
do not apply’ arguments:
This seems to be missing the spirit, and the law is, to my limited
knowledge, not a deterministic automaton with an exact mathematical
formulation without any bit flips.
Greetings,
Maxime.
(*) The BSD modules are presumably fine though (unverified)! But Guix
does _not_ (currently) support BSDs.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-03-17 17:22 ` Maxime Devos
@ 2022-03-17 18:38 ` zimoun
2022-03-17 19:10 ` Maxime Devos
2022-03-19 14:24 ` raid5atemyhomework via Guix-patches via
1 sibling, 1 reply; 81+ messages in thread
From: zimoun @ 2022-03-17 18:38 UTC (permalink / raw)
To: Maxime Devos
Cc: Liliana Marie Prikler, raid5atemyhomework, 45692@debbugs.gnu.org
Hi Maxime,
On Thu, 17 Mar 2022 at 18:23, Maxime Devos <maximedevos@telenet.be> wrote:
> I have been more-or-less ignoring the ZFS patches since some time after
> <https://issues.guix.gnu.org/45692#44>. If ZFS people(^), after a
> disagreement about licensing concerns, directly jump to accusations of
> gaslighting and sabogate, completely ignoring my previous arguments (*)
> without trying to refute any of them or bringing new arguments, then I
> don't want to be involved with ZFS.
I sympathize with you Maxime.
> Also, the various ‘work-arounds’ around the GPL<->CDLL incompatibility
> still seem super fishy to me even if they _might_ be technically
> correct. To me, this makes reviewing the code practically pointless --
> why review the zfs service patches if they will have to be reverted
> due to incompatibility concerns anyway? Summarised:
You wrote:
To be clear, I don't oppose the inclusion of the ZFS service on
basis of the licensing anymore.
<https://issues.guix.gnu.org/45692#57>
an your accurate summary gives me the impression you changed your
mind. Just to be sure, did you ?
Cheers,
simon
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-03-17 18:38 ` zimoun
@ 2022-03-17 19:10 ` Maxime Devos
0 siblings, 0 replies; 81+ messages in thread
From: Maxime Devos @ 2022-03-17 19:10 UTC (permalink / raw)
To: zimoun; +Cc: Liliana Marie Prikler, raid5atemyhomework, 45692@debbugs.gnu.org
[-- Attachment #1: Type: text/plain, Size: 568 bytes --]
zimoun schreef op do 17-03-2022 om 19:38 [+0100]:
> You wrote:
>
> To be clear, I don't oppose the inclusion of the ZFS service on
> basis of the licensing anymore.
>
> <https://issues.guix.gnu.org/45692#57>
>
> an your accurate summary gives me the impression you changed your
> mind. Just to be sure, did you ?
Yes, indeed. IIRC, I found myself often changing my mind one way or
the other quite often back then, seems like currently it's back to
opposal on basis of licensing.
Greetings,
Maxime.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-03-17 8:24 ` Liliana Marie Prikler
2022-03-17 17:22 ` Maxime Devos
@ 2022-03-19 14:09 ` raid5atemyhomework via Guix-patches via
2022-03-19 16:22 ` Leo Famulari
1 sibling, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-03-19 14:09 UTC (permalink / raw)
To: Liliana Marie Prikler; +Cc: Maxime Devos, 45692@debbugs.gnu.org
Good morning Liliana,
> Why is it necessary to define a
> file system as a service?
Because getting ZFS to work on Linux requires:
* That the ZFS module be added to the `initrd` so that Linux-libre can load it.
* That the ZFS module be actually *loaded*, because otherwise the system assumes it is loaded on some `udev` trigger.
* That the ZFS module is informed of when it should start scanning for ZFS pools in the system.
* That the above step happens before `user-processes` is started.
All that additonal complexity is conveniently packaged in the Guix service system, and you could have learned that if you had just bothered to actually read the patch.
+ (list ;; Install OpenZFS kernel module into kernel profile.
+ (service-extension linux-loadable-module-service-type
+ zfs-loadable-modules)
+ ;; And load it.
+ (service-extension kernel-module-loader-service-type
+ (const '("zfs")))
+ ;; Make sure ZFS pools and datasets are mounted at
+ ;; boot.
+ (service-extension shepherd-root-service-type
+ zfs-shepherd-services)
+ ;; Make sure user-processes don't start until
+ ;; after ZFS does.
+ (service-extension user-processes-service-type
+ zfs-user-processes)
+ ;; Install automated scrubbing and snapshotting.
+ (service-extension mcron-service-type
+ zfs-mcron-jobs)
+
+ ;; Install ZFS management commands in the system
+ ;; profile.
+ (service-extension profile-service-type
+ (compose list make-zfs-package))
+ ;; Install ZFS udev rules.
+ (service-extension udev-service-type
+ (compose list make-zfs-package))))
> Why do we need to export a seemingly
> unrelated variable?
If you are referring to the `dependency->shepherd-service-name` variable, it is necessary in order to defer starting of ZFS pool scanning to after all `mapped-device` dependencies have been opened.
> Can this be tested? Is this sufficiently tested?
Yes, I have run VMs on each version I have ever sent.
> Are there any points Maxime that were drowned out by a huge wall of
> licensing-related messages being passed back and forth that I will not
> attempt to sift through in order to respond to this message? If so,
> have those been sufficiently addressed?
Other than grammar and wording of documentation / comments, and one point about using Guix-style Scheme instead of a bit of shell (even though there are probably more people who can properly review the latter than the former), Maxime had no other points.
Other reviewers had and those were already addressed.
>
> Another complicating factor for this bug in particular is that the mumi
> web interface and the raw messages are out of sync; I have no idea why
> that is the case, but trying to fetch a patch only to get one of your
> bump messages is not particularly encouraging.
Oh come on stop your shitty excuses. Nobody would look at this unless I showed up at least once a week to pester Guix maintainers. I already tried the wait-for-people-and-they-will-come. Squeaky wheel gets the grease. If you want contributors to be more respectful, then put up definitive responsibilities of who to contact for what and what to do if patches are being ignored. Otherwise I am just going to talk smack at you, Prikler.
NO Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-03-17 17:22 ` Maxime Devos
2022-03-17 18:38 ` zimoun
@ 2022-03-19 14:24 ` raid5atemyhomework via Guix-patches via
2022-03-20 4:42 ` Maxim Cournoyer
1 sibling, 1 reply; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-03-19 14:24 UTC (permalink / raw)
To: Maxime Devos; +Cc: 45692@debbugs.gnu.org
Hello Maxime,
> - That seems rather inconvenient, why not use BTRFS instead which
> seems quite capable and doesn't have this weird restriction?
BTRFS IS NOT CAPABLE.
Did you notice my pseudonym? "`raid5` ate my homework". I used the BTRFS `raid5` mode, once. It LOST MY DATA. Never again. ZFS supports RAIDZ1 and has not lost my data at all yet. I've replaced ZFS disks on my pool. No data loss. It keeps on going on.
A file system that loses data is not a file system. It is a disaster.
BTRFS is not an acceptable substitute for ZFS.
If ZFS is removed from Guix, I am switching to Ubuntu and keeping my ZFS pool, I am not going to switch to BTRFS just to keep running Guix, I would *like* to run only fully-free software, especially since I took the trouble of paying a premium for a server that had coreboot, but my data is more important and BTRFS is not an acceptable substitute for ZFS.
The only restriction needed is to prevent binary redistribution. Yes, I agree it is inconvenient to always have to transfer source code and recompile each time. But it is a ***lot*** more inconvenient to replace my lost data because BTRFS couldn't cut it despite more than a decade of development. At least I can re-download the source code for ZFS each time from many trivial sources. My `/home`, I cannot. That is a bigger inconvenience.
Thanks
raid5atemyhomework
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] (No Subject)
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
` (11 preceding siblings ...)
2022-01-07 4:21 ` [bug#45692] " raid5atemyhomework via Guix-patches via
@ 2022-03-19 14:25 ` raid5atemyhomework via Guix-patches via
12 siblings, 0 replies; 81+ messages in thread
From: raid5atemyhomework via Guix-patches via @ 2022-03-19 14:25 UTC (permalink / raw)
To: 45692@debbugs.gnu.org
close 45692
quit
No point: nobody will review or merge anyway.
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-03-19 14:09 ` raid5atemyhomework via Guix-patches via
@ 2022-03-19 16:22 ` Leo Famulari
0 siblings, 0 replies; 81+ messages in thread
From: Leo Famulari @ 2022-03-19 16:22 UTC (permalink / raw)
To: 45692; +Cc: liliana.prikler, maximedevos
On Sat, Mar 19, 2022 at 02:09:55PM +0000, raid5atemyhomework via Guix-patches via wrote:
> Oh come on stop your shitty excuses. Nobody would look at this unless I showed up at least once a week to pester Guix maintainers. I already tried the wait-for-people-and-they-will-come. Squeaky wheel gets the grease. If you want contributors to be more respectful, then put up definitive responsibilities of who to contact for what and what to do if patches are being ignored. Otherwise I am just going to talk smack at you, Prikler.
Participating in a software project is both a technical and a social
exercise. It's not enough to merely do the technical work.
You may need to complete some smaller and less contentious contributions
along the way in order to build social trust with other people within
Guix.
There is a risk of social problems within GNU if we add ZFS to Guix:
there is not a consensus within GNU about whether it's okay to integrate
ZFS into our software.
It's important for the primary contributor of ZFS (that's you) to
demonstrate that they can manage that gracefully. None of us want to
assume the position of having to argue on your behalf.
It's true that Guix has grown large enough that formalizing teams and
points of contact for various areas would help contributors.
But until then, contributors are free to make an effort: learn who to
contact, build a collegial relationship with them, etc.
^ permalink raw reply [flat|nested] 81+ messages in thread
* [bug#45692] [PATCH 0/3] Better Support for ZFS on Guix
2022-03-19 14:24 ` raid5atemyhomework via Guix-patches via
@ 2022-03-20 4:42 ` Maxim Cournoyer
0 siblings, 0 replies; 81+ messages in thread
From: Maxim Cournoyer @ 2022-03-20 4:42 UTC (permalink / raw)
To: 45692; +Cc: raid5atemyhomework, maximedevos
Hi,
raid5atemyhomework via Guix-patches via <guix-patches@gnu.org> writes:
> Hello Maxime,
>
>> - That seems rather inconvenient, why not use BTRFS instead which
>> seems quite capable and doesn't have this weird restriction?
>
> BTRFS IS NOT CAPABLE.
>
> Did you notice my pseudonym? "`raid5` ate my homework". I used the
> BTRFS `raid5` mode, once. It LOST MY DATA. Never again. ZFS
> supports RAIDZ1 and has not lost my data at all yet. I've replaced
> ZFS disks on my pool. No data loss. It keeps on going on.
>
> A file system that loses data is not a file system. It is a disaster.
>
>
> BTRFS is not an acceptable substitute for ZFS.
>
> If ZFS is removed from Guix, I am switching to Ubuntu and keeping my
> ZFS pool, I am not going to switch to BTRFS just to keep running Guix,
> I would *like* to run only fully-free software, especially since I
> took the trouble of paying a premium for a server that had coreboot,
> but my data is more important and BTRFS is not an acceptable
> substitute for ZFS.
Btrfs RAID5 or RAID6 having a write hole leading to potential data loss
upon hard reset has been a known issue for like a decade, and nobody has
worked on improving that [0]. RAID10 is fine though, and so is RAID1 or
RAID0. I've used it (Btrfs RAID1 with zstd compression) for years on
various Guix systems without any issue.
> The only restriction needed is to prevent binary redistribution. Yes,
> I agree it is inconvenient to always have to transfer source code and
> recompile each time. But it is a ***lot*** more inconvenient to
> replace my lost data because BTRFS couldn't cut it despite more than a
> decade of development. At least I can re-download the source code for
> ZFS each time from many trivial sources. My `/home`, I cannot. That
> is a bigger inconvenience.
With my personal experience suggesting that Btrfs is a solid file
system, I respectfully disagree :-). At any rate, don't forget to
backup your precious data to somewhere safe; as RAID is no substitute
(ever had a PSU failure blowing up multiple components?).
Thanks,
Maxim
[0] https://btrfs.wiki.kernel.org/index.php/Status#RAID56
^ permalink raw reply [flat|nested] 81+ messages in thread
end of thread, other threads:[~2022-03-20 4:44 UTC | newest]
Thread overview: 81+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-01-06 15:52 [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix raid5atemyhomework via Guix-patches via
2021-01-06 15:54 ` [bug#45692] [PATCH 1/4] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
2021-01-08 16:16 ` raid5atemyhomework via Guix-patches via
2021-02-10 14:13 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
2021-02-10 15:44 ` Ludovic Courtès
2021-02-10 16:49 ` raid5atemyhomework via Guix-patches via
2021-01-06 15:55 ` [bug#45692] [PATCH 2/4] gnu: Make file-systems target extensible by services raid5atemyhomework via Guix-patches via
2021-01-23 13:05 ` 宋文武
2021-01-25 0:18 ` guix-patches--- via
2021-02-10 14:17 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
2021-02-10 14:46 ` raid5atemyhomework via Guix-patches via
2021-01-06 15:56 ` [bug#45692] [PATCH 3/4] gnu: Fix ZFS package raid5atemyhomework via Guix-patches via
2021-01-07 8:23 ` Danny Milosavljevic
2021-01-06 15:57 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
2021-01-06 19:41 ` [bug#45703] kernel-module-configuration-service for configuring kernel parameters Danny Milosavljevic
2021-01-07 0:04 ` raid5atemyhomework via Guix-patches via
2021-01-07 5:38 ` [bug#45692] " raid5atemyhomework via Guix-patches via
2021-01-07 9:16 ` [bug#42193] " raid5atemyhomework via Guix-patches via
2021-01-08 15:02 ` [bug#45692] [PATCH 4/4] gnu: Add ZFS service raid5atemyhomework via Guix-patches via
2021-01-09 8:31 ` raid5atemyhomework via Guix-patches via
2021-02-08 3:31 ` Danny Milosavljevic
2021-02-08 6:25 ` raid5atemyhomework via Guix-patches via
2021-02-10 14:27 ` [bug#45692] [PATCH 0/4] Even Better ZFS Support on Guix Ludovic Courtès
2021-02-10 14:32 ` raid5atemyhomework via Guix-patches via
2021-02-13 1:49 ` raid5atemyhomework via Guix-patches via
2021-03-22 14:33 ` [bug#45692] [PATCH v3 0/3] New patch series for " raid5atemyhomework via Guix-patches via
2021-03-28 12:55 ` Léo Le Bouter via Guix-patches via
2021-03-29 4:39 ` raid5atemyhomework via Guix-patches via
2021-07-23 15:11 ` raid5atemyhomework via Guix-patches via
2021-03-22 14:33 ` [bug#45692] [PATCH v3 1/3] gnu: Allow services to install kernel-loadable modules raid5atemyhomework via Guix-patches via
2021-05-11 14:17 ` Danny Milosavljevic
2021-03-22 14:34 ` [bug#45692] [PATCH v3 2/3] gnu: Add zfs-auto-snapshot raid5atemyhomework via Guix-patches via
2021-05-11 14:05 ` Danny Milosavljevic
2021-05-13 1:21 ` raid5atemyhomework via Guix-patches via
2021-05-13 13:08 ` Danny Milosavljevic
2021-03-22 14:35 ` [bug#45692] [PATCH v3 3/3] gnu: Add ZFS service type raid5atemyhomework via Guix-patches via
2021-07-25 14:03 ` raid5atemyhomework via Guix-patches via
2021-07-25 14:31 ` [bug#45692] [PATCH v4 " raid5atemyhomework via Guix-patches via
2021-08-01 9:41 ` raid5atemyhomework via Guix-patches via
2021-08-10 11:43 ` raid5atemyhomework via Guix-patches via
2021-08-31 0:48 ` raid5atemyhomework via Guix-patches via
2021-09-02 20:57 ` Maxime Devos
2021-09-02 22:22 ` Maxime Devos
2021-09-03 12:41 ` raid5atemyhomework via Guix-patches via
2021-09-04 18:58 ` raid5atemyhomework via Guix-patches via
2021-09-06 8:08 ` zimoun
2021-09-06 10:40 ` Maxime Devos
2021-09-06 11:08 ` raid5atemyhomework via Guix-patches via
2021-09-06 17:17 ` zimoun
2021-09-07 9:54 ` Maxime Devos
2021-09-08 1:23 ` raid5atemyhomework via Guix-patches via
2021-09-15 14:04 ` raid5atemyhomework via Guix-patches via
2021-09-21 9:42 ` zimoun
2021-09-04 21:19 ` Xinglu Chen
2021-09-06 10:52 ` raid5atemyhomework via Guix-patches via
2021-09-06 14:22 ` Xinglu Chen
2021-09-02 21:24 ` [bug#45692] Gaslighting Mason Loring Bliss
2021-09-03 12:22 ` Maxime Devos
2021-09-06 7:59 ` [bug#45692] zimoun
2021-09-30 14:56 ` [bug#45692] [PATCH v5 3/3] gnu: Add ZFS service type raid5atemyhomework via Guix-patches via
2021-10-19 13:18 ` raid5atemyhomework via Guix-patches via
2021-10-27 7:30 ` raid5atemyhomework via Guix-patches via
2021-10-27 16:38 ` pelzflorian (Florian Pelz)
2021-11-30 15:26 ` raid5atemyhomework via Guix-patches via
2021-12-12 13:32 ` raid5atemyhomework via Guix-patches via
2021-12-21 21:15 ` [bug#45643] [PATCH 0/3] Better Support for ZFS on Guix Brice Waegeneire
2022-01-01 11:59 ` [bug#45692] bug#45643: " raid5atemyhomework via Guix-patches via
2022-01-19 14:24 ` raid5atemyhomework via Guix-patches via
2022-01-07 4:21 ` [bug#45692] " raid5atemyhomework via Guix-patches via
2022-02-14 14:10 ` raid5atemyhomework via Guix-patches via
2022-02-18 7:13 ` raid5atemyhomework via Guix-patches via
2022-03-16 23:44 ` raid5atemyhomework via Guix-patches via
2022-03-17 8:24 ` Liliana Marie Prikler
2022-03-17 17:22 ` Maxime Devos
2022-03-17 18:38 ` zimoun
2022-03-17 19:10 ` Maxime Devos
2022-03-19 14:24 ` raid5atemyhomework via Guix-patches via
2022-03-20 4:42 ` Maxim Cournoyer
2022-03-19 14:09 ` raid5atemyhomework via Guix-patches via
2022-03-19 16:22 ` Leo Famulari
2022-03-19 14:25 ` [bug#45692] (No Subject) raid5atemyhomework via Guix-patches via
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.