[bug#45643] [PATCH 0/3] Better Support for ZFS on Guix

[bug#45643] [PATCH 0/3] Better Support for ZFS on Guix

[raid5atemyhomework via Guix-patches via]

ZFS package is broken, and we should really recompile ZFS at each kernel update anyway.

This patchset includes:

* Patch 1:
  * Fixes for "util-linux" dependency which causes problems with mounting ZFS datasets.
  * Expose a "make-zfs-package" procedure to create kernel-specific ZFS packages as suggested here: https://issues.guix.gnu.org/45403#3
  * Add a service "zfs-loader-service-type" that provisions "zfs-loader", that modprobes ZFS and automounts ZFS datasets at bootup. The latter behavior is what is expected of ZFS from other OSs, so it's best to imitate that as well.
  * Document how to set up and use the above.
  * This makes ZFS on Guix actually *possible*.  Tested in a VM.
* Patch 2:
  * Make the "file-systems" provisioning service a target similar to "user-processes", so that other services can insert themselves into "file-systems" mounting.
  * Make the "zfs-loader" part of the "file-systems" target if it is instantiated.
  * This enables /home on ZFS (previous patch is not enough as "user-homes" service is part of the "user-processes"; we need to make ZFS loading and mounting earlier).
* Patch 3
  * Disable checking of "device" field for declared file-systems that are of type "zfs".
  * Make file-systems of type "zfs" dependent on "zfs-loader" to ensure that "zfs" module is loaded and all pools are imported and known by the ZFS module.
  * This enables declaring ZFS datasets as (file-system ...) declarations, if the sysad wants to explicitly document the mountpoints in the configuration file rather than having ZFS manage mountpoints.

[bug#45643] [PATCH 1/3] gnu, doc: Create and document procedure to compile ZFS for specific kernel.

[raid5atemyhomework via Guix-patches via]

From 004a84a38266eb8735f1e127f5794bf6d66b9272 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Fri, 1 Jan 2021 20:26:42 +0800
Subject: [PATCH 1/3] gnu, doc: Create and document procedure to compile ZFS
 for specific kernel.

---
 doc/guix.texi                 | 100 ++++++++++++++++++++++++++++++++++
 gnu/local.mk                  |   1 +
 gnu/packages/file-systems.scm |  25 ++++++++-
 gnu/services/file-systems.scm |  72 ++++++++++++++++++++++++
 4 files changed, 196 insertions(+), 2 deletions(-)
 create mode 100644 gnu/services/file-systems.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index 1081ed26a3..5ad3907dbe 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13885,6 +13885,106 @@ a file system declaration such as:
 compress-force=zstd,space_cache=v2"))
 @end lisp

+@node ZFS file system
+@subsection ZFS file system
+
+The ZFS on Linux file system cannot legally be downloaded as part of the
+Linux kernel, but you as a user can do anything on hardware you own,
+including download ZFS as source code, compile ZFS as a kernel module,
+and link it into Linux.
+
+As a large and complex kernel module, ZFS has to be compiled for a
+specific version of Linux. Often the latest ZFS package available in Guix
+cannot be compiled with the latest Linux kernel available in Guix, so
+installing the @code{zfs} package in your system configuration file will
+fail.
+
+Instead, you have to check the
+@url{https://github.com/openzfs/zfs/releases,OpenZFS release notes} for
+the specific version of ZFS that Guix has packaged to determine what
+Linux kernels you can use, then check the @code{linux-libre} packages
+that Guix has packaged, and select one you can use on your system.
+
+Then, you have to modify your system configuration file, and create a
+ZFS package that compiles using the specific Linux version you chose.
+Below is a sketch of how you would modify your @code{operating-system}
+declaration in order to set up ZFS:
+
+@lisp
+(use-modules (gnu))
+;; @dots{}
+(use-service-modules
+  ; @dots{}
+  file-systems)
+(use-package-modules
+  ; @dots{}
+  linux
+  file-systems)
+
+;; @dots{}
+
+;;; (1) Select a specific kernel.
+(define system-kernel linux-libre-5.4)
+;;; (2) Define a ZFS package for your kernel.
+(define system-zfs (make-zfs-package system-kernel))
+
+;; @dots{}
+
+(operating-system
+  ;;; (3) Specify your selected kernel.
+  (kernel system-kernel)
+  ;;; (4) Add the "module" output of the system ZFS package to
+  ;;; the kernel-loadable modules.
+  (kernel-loadable-modules (list (list system-zfs "module")))
+
+  ;; @dots{}
+
+  (packages
+    ;;; (5) Add the system ZFS package to global packages so that
+    ;;; "zfs", "zpool" etc. commands are available.
+    (cons* system-zfs
+           ; @dots{}
+           %base-packages))
+
+  ;; @dots{}
+
+  (services
+    ;;; (6) Add a service that loads ZFS and all ZFS pools at boot.
+    (cons* (service zfs-loader-service-type
+                    system-zfs)
+           ; @dots{}
+           %base-services))
+  ;; @dots{}
+  )
+@end lisp
+
+@deffn (Scheme Procedure) make-zfs-package @var{kernel}
+This procedure creates a package which, when included as a package
+in your system, can be loaded as a kernel module for the specified
+@var{kernel}, a Linux kernel package.
+It is intended to create a system-specific ZFS kernel module for
+the Linux kernel you will use in your system.
+@end deffn
+
+@deffn (Scheme Variable) zfs-loader-service-type
+This is the service that loads the ZFS kernel module and imports all
+ZFS pools. The value associated with @code{zfs-loader-service-type}
+services must be the ZFS package you defined for your system, and
+whose @code{"module"} output you have included as a loadable kernel
+module.
+@end deffn
+
+When you have modified your @code{operating-system} to include ZFS
+as in the above and reconfigured and rebooted, you will now be able
+to create, mount, and manage ZFS pools. At each boot, ZFS will
+automatically open any ZFS pools it detects in your system, and will
+mount ZFS pools that have a non-@code{legacy} mountpoint. If ZFS
+finds mountable ZFS filesystems that are encrypted by passphrase, it
+will prompt for passphrases on the console.
+
+ZFS as root filesystem is not supported yet. ZFS for @code{/home} is
+also probably not easily doable yet.
+
 @node Mapped Devices
 @section Mapped Devices

diff --git a/gnu/local.mk b/gnu/local.mk
index c03a8b9f51..e59a3bb9f7 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -601,6 +601,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/services/dict.scm				\
   %D%/services/dns.scm				\
   %D%/services/docker.scm			\
+  %D%/services/file-systems.scm			\
   %D%/services/authentication.scm		\
   %D%/services/games.scm			\
   %D%/services/ganeti.scm			\
diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 895ad069c5..e8f49a59fa 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -944,16 +944,37 @@ APFS.")
        ("openssl" ,openssl)
        ("python" ,python)
        ("python-cffi" ,python-cffi)
-       ("util-linux" ,util-linux "lib")
+       ("util-linux" ,util-linux)
+       ("util-linux-lib" ,util-linux "lib")
        ("zlib" ,zlib)))
     (home-page "https://zfsonlinux.org/")
     (synopsis "Native ZFS on Linux")
     (description
      "ZFS on Linux is an advanced file system and volume manager which was
 originally developed for Solaris and is now maintained by the OpenZFS
-community.")
+community.
+
+DO NOT INSTALL THIS PACKAGE. Instead, refer to the 'ZFS file system' section
+of the Guix info manual for how to install ZFS.")
     (license license:cddl1.0)))

+(define-public (make-zfs-package kernel)
+  (package
+    (inherit zfs)
+    (name (string-append "zfs-for-"
+                         (package-name kernel)
+                         "-"
+                         (package-version kernel)
+                         "-version"))
+    (arguments
+      (cons* #:linux kernel (package-arguments zfs)))
+    (description
+     "ZFS on Linux is an advanced file system and volume manager which was
+originally developed for Solaris and is now maintained by the OpenZFS
+community.
+
+This package has been compiled for a specific Linux kernel.")))
+
 (define-public mergerfs
   (package
     (name "mergerfs")
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..bdc33f4028
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,72 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+  #:use-module (guix gexp)
+  #:use-module (gnu services)
+  #:use-module (gnu services shepherd))
+
+;;; ZFS
+(define (zfs-loader-shepherd-service system-zfs)
+  (let* ((zpool (file-append system-zfs "/sbin/zpool")))
+    (list
+      (shepherd-service
+       (documentation "Load ZFS kernel module and import ZFS pools.")
+       (provision '(zfs-loader))
+       (requirement '(file-systems))
+       (one-shot? #t)
+       (modules `((srfi srfi-1)
+                  (srfi srfi-34)
+                  (srfi srfi-35)
+                  (rnrs io ports)
+                  ,@%default-modules))
+       (start #~(lambda _
+                  (if (not (file-exists? "/proc/sys/kernel/modprobe"))
+                      (begin (format (current-error-port) "error: ~a~%"
+                                     "Kernel is missing loadable module support.")
+                             #f)
+                      (and
+                        (let ((modprobe (call-with-input-file
+                                          "/proc/sys/kernel/modprobe" get-line)))
+                          (guard (c ((message-condition? c)
+                                     (format (current-error-port)
+                                             "error loading 'zfs' kernel module: ~a~%"
+                                             (condition-message c))
+                                     #f))
+                            (invoke/quiet modprobe "--" "zfs")))
+                        (guard (c ((message-condition? c)
+                                   (format (current-error-port)
+                                           "error importing zpools: ~a~%"
+                                           (condition-message c))
+                                   #f))
+                          ;; 'current-output-port' is typically connected to /dev/klog
+                          ;; in PID 1, so redirect it to the console so we are talking
+                          ;; to the user if zpool sees an encrpyted fs that needs a
+                          ;; passphrase.
+                          (with-output-to-port (current-error-port)
+                            (lambda ()
+                              (invoke #$zpool "import" "-a" "-l"))))))))))))
+
+(define-public zfs-loader-service-type
+  (service-type
+   (name 'zfs-loader)
+   (description "Load ZFS kernel module and import ZFS pools.")
+   (extensions
+    (list (service-extension shepherd-root-service-type zfs-loader-shepherd-service)
+          (service-extension user-processes-service-type (const '(zfs-loader)))))))
+
--
2.29.2

[bug#45643] [PATCH 2/3] gnu: Make 'file-systems' target extensible.

[raid5atemyhomework via Guix-patches via]

From fc4538963960550b678713ec46ac461cfbd6173e Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Sun, 3 Jan 2021 21:09:02 +0800
Subject: [PATCH 2/3] gnu: Make 'file-systems' target extensible.

This is to support something like ZFS, which has a service that mounts
ZFS file systems and does not normally use /etc/fstab
---
 doc/guix.texi                 |  9 +++++++--
 gnu/services/base.scm         | 37 +++++++++++++++++++++++++----------
 gnu/services/file-systems.scm |  5 +++--
 3 files changed, 37 insertions(+), 14 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5ad3907dbe..4544b481b0 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13982,8 +13982,13 @@ mount ZFS pools that have a non-@code{legacy} mountpoint. If ZFS
 finds mountable ZFS filesystems that are encrypted by passphrase, it
 will prompt for passphrases on the console.

-ZFS as root filesystem is not supported yet. ZFS for @code{/home} is
-also probably not easily doable yet.
+You can put @code{/home} on a ZFS filesystem by setting it as the
+mountpoint of some ZFS filesystem. However, ZFS will refuse to mount
+onto a non-empty directory, so if you already have an existing
+@code{/home} you have to remove all files in it, then reboot to let
+ZFS mount into @code{/home}.
+
+ZFS as root filesystem is not supported yet.

 @node Mapped Devices
 @section Mapped Devices
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 945b546607..deffd49154 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -66,6 +66,7 @@
                %default-substitute-urls)
   #:export (fstab-service-type
             root-file-system-service
+            file-systems-service-type
             file-system-service-type
             swap-service
             host-name-service
@@ -362,18 +363,30 @@ FILE-SYSTEM."
                        (gnu system file-systems)
                        ,@%default-modules)))))))

+(define (file-systems-service requirements)
+  (list
+    (shepherd-service
+     (provision '(file-systems))
+     (requirement (cons* 'root-file-system 'user-file-systems requirements))
+     (documentation "Target for all the initially-mounted file systems")
+     (start #~(const #t))
+     (stop #~(const #t)))))
+
+(define file-systems-service-type
+  (service-type
+    (name 'file-systems)
+    (extensions (list (service-extension shepherd-root-service-type
+                                         file-systems-service)))
+    (compose concatenate)
+    (extend append)
+    ;; Extensions can add new services to this list.
+    (default-value '())
+    (description "The @code{file-systems} service is the target that is started
+when all file systems have been mounted.")))
+
 (define (file-system-shepherd-services file-systems)
   "Return the list of Shepherd services for FILE-SYSTEMS."
   (let* ((file-systems (filter file-system-mount? file-systems)))
-    (define sink
-      (shepherd-service
-       (provision '(file-systems))
-       (requirement (cons* 'root-file-system 'user-file-systems
-                           (map file-system->shepherd-service-name
-                                file-systems)))
-       (documentation "Target for all the initially-mounted file systems")
-       (start #~(const #t))
-       (stop #~(const #f))))

     (define known-mount-points
       (map file-system-mount-point file-systems))
@@ -403,7 +416,7 @@ FILE-SYSTEM."
                            (filter (negate known?) (mount-points)))
                  #f))))

-    (cons* sink user-unmount
+    (cons* user-unmount
            (map file-system-shepherd-service file-systems))))

 (define (file-system-fstab-entries file-systems)
@@ -431,6 +444,10 @@ FILE-SYSTEM."
                        (service-extension fstab-service-type
                                           file-system-fstab-entries)

+                       ;; Have 'file-systems' depend on each file-system
+                       (service-extension file-systems-service-type
+                                          (cut map file-system->shepherd-service-name <>))
+
                        ;; Have 'user-processes' depend on 'file-systems'.
                        (service-extension user-processes-service-type
                                           (const '(file-systems)))))
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
index bdc33f4028..73a6d52538 100644
--- a/gnu/services/file-systems.scm
+++ b/gnu/services/file-systems.scm
@@ -19,6 +19,7 @@
 (define-module (gnu services file-systems)
   #:use-module (guix gexp)
   #:use-module (gnu services)
+  #:use-module (gnu services base)
   #:use-module (gnu services shepherd))

 ;;; ZFS
@@ -28,7 +29,7 @@
       (shepherd-service
        (documentation "Load ZFS kernel module and import ZFS pools.")
        (provision '(zfs-loader))
-       (requirement '(file-systems))
+       (requirement '(root-file-system))
        (one-shot? #t)
        (modules `((srfi srfi-1)
                   (srfi srfi-34)
@@ -68,5 +69,5 @@
    (description "Load ZFS kernel module and import ZFS pools.")
    (extensions
     (list (service-extension shepherd-root-service-type zfs-loader-shepherd-service)
-          (service-extension user-processes-service-type (const '(zfs-loader)))))))
+          (service-extension file-systems-service-type (const '(zfs-loader)))))))

--
2.29.2

[bug#45643] [PATCH 3/3] gnu, guix: Support mounting legacy-mounted ZFS in operating-system form.

[raid5atemyhomework via Guix-patches via]

From 59c9bd5642e33962798c01f4dcf30be38ead4ab8 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Mon, 4 Jan 2021 08:22:01 +0800
Subject: [PATCH 3/3] gnu, guix: Support mounting legacy-mounted ZFS in
 operating-system form.

---
 doc/guix.texi              | 18 ++++++++++++++++++
 gnu/build/file-systems.scm |  2 ++
 gnu/machine/ssh.scm        |  3 +++
 gnu/services/base.scm      |  5 ++++-
 guix/scripts/system.scm    |  3 +++
 5 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 4544b481b0..2909674302 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13982,6 +13982,24 @@ mount ZFS pools that have a non-@code{legacy} mountpoint. If ZFS
 finds mountable ZFS filesystems that are encrypted by passphrase, it
 will prompt for passphrases on the console.

+You should @emph{not} put ZFS datasets with a non-@code{legacy} mountpoint
+into the @code{file-systems} field of your @code{operating-sstem}. However,
+if you want to declare them in your @code{file-systems} field, you can
+set their ZFS mountpoints to @code{legacy} and add a @code{file-system}
+declaration for them, using the ZFS pool dataset name as the @code{device}
+field:
+
+@example
+zfs set mountpoint=legacy pool/filesystem
+@end example
+
+@lisp
+(file-system
+  (device "pool/filesystem")
+  (mount-point "/path/to/mountpoint")
+  (type "zfs"))
+@end lisp
+
 You can put @code{/home} on a ZFS filesystem by setting it as the
 mountpoint of some ZFS filesystem. However, ZFS will refuse to mount
 onto a non-empty directory, so if you already have an existing
diff --git a/gnu/build/file-systems.scm b/gnu/build/file-systems.scm
index ddf6117b67..af75aee2b6 100644
--- a/gnu/build/file-systems.scm
+++ b/gnu/build/file-systems.scm
@@ -828,6 +828,8 @@ containing ':/')."
      ((string-prefix? "f2fs" type) check-f2fs-file-system)
      ((string-prefix? "ntfs" type) check-ntfs-file-system)
      ((string-prefix? "nfs" type) (const 'pass))
+     ;; "ZFS doesn't need fsck."
+     ((string-prefix? "zfs" type) (const 'pass))
      (else #f)))

   (if check-procedure
diff --git a/gnu/machine/ssh.scm b/gnu/machine/ssh.scm
index 08c653ba17..cdd8913a00 100644
--- a/gnu/machine/ssh.scm
+++ b/gnu/machine/ssh.scm
@@ -175,6 +175,9 @@ exist on the machine."
                                 %pseudo-file-system-types))
                    ;; Don't try to validate network file systems.
                    (not (string-prefix? "nfs" (file-system-type fs)))
+                   ;; Don't try to validate ZFS file systems.
+                   ;; XXX We should validate these by 'zfs status'.
+                   (not (string-prefix? "zfs" (file-system-type fs)))
                    (not (memq 'bind-mount (file-system-flags fs)))))
             (operating-system-file-systems (machine-operating-system machine))))

diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index deffd49154..dcacc5b4b8 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -322,7 +322,10 @@ FILE-SYSTEM."
            (shepherd-service
             (provision (list (file-system->shepherd-service-name file-system)))
             (requirement `(root-file-system udev
-                           ,@(map dependency->shepherd-service-name dependencies)))
+                           ,@(map dependency->shepherd-service-name dependencies)
+                           ,@(if (string-prefix? "zfs" (file-system-type file-system))
+                                 '(zfs-loader)
+                                 '())))
             (documentation "Check, mount, and unmount the given file system.")
             (start #~(lambda args
                        #$(if create?
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index 0dcf2b3afe..1515062ada 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -564,6 +564,9 @@ any, are available.  Raise an error if they're not."
                                 %pseudo-file-system-types))
                    ;; Don't try to validate network file systems.
                    (not (string-prefix? "nfs" (file-system-type fs)))
+                   ;; Don't try to validate ZFS file systems.
+                   ;; XXX We should validate these by 'zfs status'.
+                   (not (string-prefix? "zfs" (file-system-type fs)))
                    (not (memq 'bind-mount (file-system-flags fs)))))
             file-systems))

--
2.29.2