unofficial mirror of guix-patches@gnu.org 
 help / color / mirror / code / Atom feed
* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
@ 2022-05-02 19:11 Brian Cully via Guix-patches via
  2022-05-02 22:03 ` Maxime Devos
                   ` (9 more replies)
  0 siblings, 10 replies; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-05-02 19:11 UTC (permalink / raw)
  To: 55231

This patch allows copying of out-of-tree kernel modules to the Linux
initrd.

For out-of-tree modules to found, an extra slot has been added to
‘operating-system’: ‘initrd-extra-module-paths’, which contains a list
containing items of either FILE-LIKE or (FILE-LIKE OUTPUT) which will be
searched for modules in addition to the standard Linux module path. The
required modules can then be added to ‘initrd-modules’ as normal and all paths
will be searched for it, including for any modules depended on.

* gnu/build/linux-modules.scm (find-module-file): change DIRECTORY argument to
DIRECTORIES. Now takes a list of directories to search, rather than a single
one.
* gnu/system.scm <operating-system>: Add INITRD-EXTRA-MODULE-PATHS
field and accessor. Takes a list of file-like objects.
* gnu/system/linux-initrd.scm (flat-linux-module-directory): change LINUX
argument to PACKAGES. Now contains a list of file-likes to search for modules.
(raw-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
to (flat-linux-extra-module-paths) along with the selected LINUX package.
(base-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
to (raw-initrd).
---
 gnu/build/linux-modules.scm | 19 ++++++++------
 gnu/system.scm              |  5 ++++
 gnu/system/linux-initrd.scm | 52 +++++++++++++++++++++++++------------
 3 files changed, 51 insertions(+), 25 deletions(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 053720574b..97b7e429ea 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -225,8 +225,8 @@ (define (file-name->module-name file)
 '.ko[.gz|.xz]' and normalizing it."
   (normalize-module-name (strip-extension (basename file))))
 
-(define (find-module-file directory module)
-  "Lookup module NAME under DIRECTORY, and return its absolute file name.
+(define (find-module-file directories module)
+  "Lookup module NAME under DIRECTORIES, and return its absolute file name.
 NAME can be a file name with or without '.ko', or it can be a module name.
 Raise an error if it could not be found.
 
@@ -247,16 +247,19 @@ (define names
                            (else chr)))
                        module))))
 
-  (match (find-files directory
-                     (lambda (file stat)
-                       (member (strip-extension
-                                (basename file)) names)))
+  (match (append-map (lambda (directory)
+                       (find-files directory
+                                   (lambda (file _stat)
+                                     (member (strip-extension
+                                              (basename file))
+                                             names))))
+                       directories)
     ((file)
      file)
     (()
-     (error "kernel module not found" module directory))
+     (error "kernel module not found" module directories))
     ((_ ...)
-     (error "several modules by that name" module directory))))
+     (error "several modules by that name" module directories))))
 
 (define* (recursive-module-dependencies files
                                         #:key (lookup-module dot-ko))
diff --git a/gnu/system.scm b/gnu/system.scm
index c3810cbeeb..15ac30c933 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -103,6 +103,7 @@ (define-module (gnu system)
             operating-system-label
             operating-system-default-label
             operating-system-initrd-modules
+            operating-system-initrd-extra-module-paths
             operating-system-initrd
             operating-system-users
             operating-system-groups
@@ -232,6 +233,8 @@ (define-record-type* <operating-system> operating-system
   (initrd-modules operating-system-initrd-modules ; list of strings
                   (thunked)                       ; it's system-dependent
                   (default %base-initrd-modules))
+  (initrd-extra-module-paths operating-system-initrd-extra-module-paths ; list of file-likes
+                             (default '()))
 
   (firmware operating-system-firmware             ; list of packages
             (default %base-firmware))
@@ -1307,6 +1310,8 @@ (define make-initrd
                #:linux (operating-system-kernel os)
                #:linux-modules
                (operating-system-initrd-modules os)
+               #:linux-extra-module-paths
+               (operating-system-initrd-extra-module-paths os)
                #:mapped-devices mapped-devices
                #:keyboard-layout (operating-system-keyboard-layout os)))
 
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 4c4c78e444..50a182d7d5 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -119,13 +119,23 @@ (define builder
                               `(#:references-graphs (("closure" ,init))))
                "/initrd.cpio.gz"))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory packages modules)
   "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+MODULES and taken from PACKAGES."
   (define imported-modules
     (source-module-closure '((gnu build linux-modules)
                              (guix build utils))))
 
+  (define (package+out->input package out)
+    (gexp-input package out))
+
+  (define package-inputs
+    (map (lambda (p)
+           (match p
+             ((p o) (package+out->input p o))
+             (p     (package+out->input p "out"))))
+         packages))
+
   (define build-exp
     (with-imported-modules imported-modules
       (with-extensions (list guile-zlib)
@@ -135,11 +145,12 @@ (define build-exp
                          (srfi srfi-1)
                          (srfi srfi-26))
 
-            (define module-dir
-              (string-append #$linux "/lib/modules"))
+            (define module-dirs
+              (map (cut string-append <> "/lib/modules")
+                   '#$package-inputs))
 
             (define modules
-              (let* ((lookup  (cut find-module-file module-dir <>))
+              (let* ((lookup  (cut find-module-file module-dirs <>))
                      (modules (map lookup '#$modules)))
                 (append modules
                         (recursive-module-dependencies
@@ -172,20 +183,23 @@ (define* (raw-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-paths '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       (helper-packages '())
                       qemu-networking?
                       volatile-root?
                       (on-error 'debug))
-  "Return as a file-like object a raw initrd, with kernel
-modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
-mounted by the initrd, possibly in addition to the root file system specified
-on the kernel command line via 'root'.  LINUX-MODULES is a list of kernel
-modules to be loaded at boot time. MAPPED-DEVICES is a list of device
-mappings to realize before FILE-SYSTEMS are mounted.
-HELPER-PACKAGES is a list of packages to be copied in the initrd. It may include
-e2fsck/static or other packages needed by the initrd to check root partition.
+  "Return as a file-like object a raw initrd, with kernel modules taken from
+LINUX.  FILE-SYSTEMS is a list of file-systems to be mounted by the initrd,
+possibly in addition to the root file system specified on the kernel command
+line via 'root'.  LINUX-MODULES is a list of kernel modules to be loaded at
+boot time. LINUX-EXTRA-MODULE-PATHS is a list of file-like objects which will
+be searched for modules in addition to the linux kernel. MAPPED-DEVICES is a
+list of device mappings to realize before FILE-SYSTEMS are mounted.
+HELPER-PACKAGES is a list of packages to be copied in the initrd. It may
+include e2fsck/static or other packages needed by the initrd to check root
+partition.
 
 When true, KEYBOARD-LAYOUT is a <keyboard-layout> record denoting the desired
 console keyboard layout.  This is done before MAPPED-DEVICES are set up and
@@ -221,7 +235,8 @@ (define file-system-scan-commands
           #~())))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory (cons linux linux-extra-module-paths)
+                                 linux-modules))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -366,6 +381,7 @@ (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-paths '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       qemu-networking?
@@ -386,9 +402,10 @@ (define* (base-initrd file-systems
 QEMU-NETWORKING? and VOLATILE-ROOT? behaves as in raw-initrd.
 
 The initrd is automatically populated with all the kernel modules necessary
-for FILE-SYSTEMS and for the given options.  Additional kernel
-modules can be listed in LINUX-MODULES.  They will be added to the initrd, and
-loaded at boot time in the order in which they appear."
+for FILE-SYSTEMS and for the given options.  Additional kernel modules can be
+listed in LINUX-MODULES.  Additional search paths for modules can be listed in
+LINUX-EXTRA-MODULE-PATHS.  They will be added to the initrd, and loaded at
+boot time in the order in which they appear."
   (define linux-modules*
     ;; Modules added to the initrd and loaded from the initrd.
     `(,@linux-modules
@@ -408,6 +425,7 @@ (define helper-packages
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules*
+              #:linux-extra-module-paths linux-extra-module-paths
               #:mapped-devices mapped-devices
               #:helper-packages helper-packages
               #:keyboard-layout keyboard-layout

base-commit: 0a64b629ae8512790d532158a72a4a25698e8157
prerequisite-patch-id: 213ce2b9743f11d45836fe0794f117f3bb84063d
-- 
2.35.1




^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
@ 2022-05-02 22:03 ` Maxime Devos
  2022-05-02 22:53   ` Brian Cully via Guix-patches via
  2022-05-13 19:46 ` Kaelyn via Guix-patches via
                   ` (8 subsequent siblings)
  9 siblings, 1 reply; 23+ messages in thread
From: Maxime Devos @ 2022-05-02 22:03 UTC (permalink / raw)
  To: Brian Cully, 55231

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

Brian Cully via Guix-patches via schreef op ma 02-05-2022 om 15:11 [-
0400]:
> This patch allows copying of out-of-tree kernel modules to the Linux
> initrd.

This needs some information in the manual -- when does the field need
to be set?  For what kernel modules?  Is this required by v4l2loopback-
linux-module and librem-ec-acpi-linux-module?  Things like that.
As-is, this functionality is hard to discover.

Greetings,
Maxime.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-02 22:03 ` Maxime Devos
@ 2022-05-02 22:53   ` Brian Cully via Guix-patches via
  0 siblings, 0 replies; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-05-02 22:53 UTC (permalink / raw)
  To: Maxime Devos; +Cc: 55231


Maxime Devos <maximedevos@telenet.be> writes:

> This needs some information in the manual -- when does the field 
> need
> to be set?  For what kernel modules?  Is this required by 
> v4l2loopback-
> linux-module and librem-ec-acpi-linux-module?  Things like that.
> As-is, this functionality is hard to discover.

I knew I missed something. How’s this look?

---
 doc/guix.texi | 42 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 34 insertions(+), 8 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5399584cb0..1ee2c1b4a3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -15173,6 +15173,16 @@ Window System.
 The list of Linux kernel modules that need to be available in the
 initial RAM disk.  @xref{Initial RAM Disk}.
 
+@item @code{initrd-extra-module-paths} (default: @code{'()})
+@cindex initrd
+@cindex initial RAM disk
+A list of paths outside of the Linux kernel tree to search for 
Linux
+kernel modules.
+
+The items in this may be either file-like objects (usually 
packages), or
+a list where the first element is a package and the second is an
+output--e.g. @code{(list (list zfs "module"))}.
+
 @item @code{initrd} (default: @code{base-initrd})
 A procedure that returns an initial RAM disk for the Linux
 kernel.  This field is provided to support low-level 
 customization and
@@ -35516,6 +35526,19 @@ file system, you would write:
   (initrd-modules (cons "megaraid_sas" %base-initrd-modules)))
 @end lisp
 
+If a module listed in @code{initrd-modules} is not included in 
the
+Linux-libre kernel, then the location to it must be added to the
+@code{initrd-extra-module-paths} list.  For example, if your root 
file
+system exists on a ZFS pool, then your configuration might look 
like the
+following:
+
+@lisp
+(operating-system
+  ;; @dots{}
+  (initrd-modules (cons "zfs" %base-initrd-modules))
+  (initrd-extra-module-paths (list (list zfs "module"))))
+@end lisp
+
 @defvr {Scheme Variable} %base-initrd-modules
 This is the list of kernel modules included in the initrd by 
 default.
 @end defvr
@@ -35629,13 +35652,15 @@ here is how to use it and customize it 
further.
 @cindex initrd
 @cindex initial RAM disk
 @deffn {Scheme Procedure} raw-initrd @var{file-systems} @
-       [#:linux-modules '()] [#:mapped-devices '()] @
-       [#:keyboard-layout #f] @
+       [#:linux-modules '()] [#:linux-extra-module-paths '()] @
+       [#:mapped-devices '()] [#:keyboard-layout #f] @
        [#:helper-packages '()] [#:qemu-networking? #f] 
        [#:volatile-root? #f]
 Return a derivation that builds a raw initrd.  @var{file-systems} 
 is
 a list of file systems to be mounted by the initrd, possibly in 
 addition to
 the root file system specified on the kernel command line via 
 @option{root}.
 @var{linux-modules} is a list of kernel modules to be loaded at 
 boot time.
+@var{linux-extra-module-paths} is a list of file-like objects to 
be searched
+for kernel modules.
 @var{mapped-devices} is a list of device mappings to realize 
 before
 @var{file-systems} are mounted (@pxref{Mapped Devices}).
 @var{helper-packages} is a list of packages to be copied in the 
 initrd.
@@ -35660,12 +35685,13 @@ to it are lost.
 @deffn {Scheme Procedure} base-initrd @var{file-systems} @
        [#:mapped-devices '()] [#:keyboard-layout #f] @
        [#:qemu-networking? #f] [#:volatile-root? #f] @
-       [#:linux-modules '()]
-Return as a file-like object a generic initrd, with kernel
-modules taken from @var{linux}.  @var{file-systems} is a list of 
 file-systems to be
-mounted by the initrd, possibly in addition to the root file 
 system specified
-on the kernel command line via @option{root}. 
 @var{mapped-devices} is a list of device
-mappings to realize before @var{file-systems} are mounted.
+       [#:linux-modules '()] [#:linux-extra-module-paths '()]
+Return as a file-like object a generic initrd, with kernel 
modules taken
+from @var{linux} and @var{linux-extra-module-paths}. 
@var{file-systems}
+is a list of file-systems to be mounted by the initrd, possibly 
in
+addition to the root file system specified on the kernel command 
line
+via @option{root}.  @var{mapped-devices} is a list of device 
mappings to
+realize before @var{file-systems} are mounted.
 
 When true, @var{keyboard-layout} is a @code{<keyboard-layout>} 
 record denoting
 the desired console keyboard layout.  This is done before 
 @var{mapped-devices}
-- 
2.35.1




^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
  2022-05-02 22:03 ` Maxime Devos
@ 2022-05-13 19:46 ` Kaelyn via Guix-patches via
  2022-05-13 20:25 ` Maxime Devos
                   ` (7 subsequent siblings)
  9 siblings, 0 replies; 23+ messages in thread
From: Kaelyn via Guix-patches via @ 2022-05-13 19:46 UTC (permalink / raw)
  To: 55231@debbugs.gnu.org

I've read through the original patch and the doc patch, and based on my own tinkering with guix initrd generation, the patches look good to me. I'm also happy to see the functionality being added, and already have plans for using it with zfs (to further the goal of an encrypted multi-disk zfs root).

Cheers,
Kaelyn




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
  2022-05-02 22:03 ` Maxime Devos
  2022-05-13 19:46 ` Kaelyn via Guix-patches via
@ 2022-05-13 20:25 ` Maxime Devos
  2022-05-21 19:12 ` [bug#55231] [PATCH v2] " Brian Cully via Guix-patches via
                   ` (6 subsequent siblings)
  9 siblings, 0 replies; 23+ messages in thread
From: Maxime Devos @ 2022-05-13 20:25 UTC (permalink / raw)
  To: 55231

[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]



(for whatever reason, looks like this didn't end up in my inbox)
> +@item @code{initrd-extra-module-paths} (default: @code{'()})
> [...]
> The items in this may be either file-like objects (usually 
> packages), or
> +a list where the first element is a package and the second is an
> +output--e.g. @code{(list (list zfs "module"))}.


As-is, I don't think this is a good example, because
'expression->initrd' does not set #:substitutable? #false,
the 'zfs' package has the comment (*)

     `(;; The ZFS kernel module should not be downloaded since the
license
       ;; terms don't allow for distributing it, only building it
locally.
       #:substitutable? #f [...])

and IIUC, the code inside expression->initrd copies the kernel module
into
the new store item, so it looks like this accidentally suggest people
to
commit copyvios, and copyvios are currently against the law.

Greetings,
Maxime.

(*) Though I don't understand that comment: Guix is a distribution, so
by definition
it's distributing zfs -- unless it's only talking about the binaries
and not the
source code ...


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v2] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
                   ` (2 preceding siblings ...)
  2022-05-13 20:25 ` Maxime Devos
@ 2022-05-21 19:12 ` Brian Cully via Guix-patches via
  2022-06-01 15:54   ` [bug#55231] [PATCH v1] " Ludovic Courtès
  2022-05-30 20:14 ` Kaelyn via Guix-patches via
                   ` (5 subsequent siblings)
  9 siblings, 1 reply; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-05-21 19:12 UTC (permalink / raw)
  To: 55231; +Cc: Brian Cully

From: Brian Cully <bjc@kublai.com>

This patch allows copying of out-of-tree kernel modules to the Linux
initrd.

For out-of-tree modules to found, an extra slot has been added to
‘operating-system’: ‘initrd-extra-module-paths’, which contains a list
containing items of either FILE-LIKE or (FILE-LIKE OUTPUT) which will be
searched for modules in addition to the standard Linux module path. The
required modules can then be added to ‘initrd-modules’ as normal and all paths
will be searched for it, including for any modules depended on.

* doc: Update documentation for ‘initrd-extra-module-paths’.
* gnu/build/linux-modules.scm (find-module-file): change DIRECTORY argument to
DIRECTORIES. Now takes a list of directories to search, rather than a single
one.
* gnu/system.scm <operating-system>: Add INITRD-EXTRA-MODULE-PATHS
field and accessor. Takes a list of file-like objects.
* gnu/system/linux-initrd.scm (flat-linux-module-directory): change LINUX
argument to PACKAGES. Now contains a list of file-likes to search for modules.
(raw-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
to (flat-linux-extra-module-paths) along with the selected LINUX package.
(base-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
to (raw-initrd).
---
This version includes both the previous patch and the documentation.

 doc/guix.texi               | 42 ++++++++++++++++++++++++------
 gnu/build/linux-modules.scm | 19 ++++++++------
 gnu/system.scm              |  5 ++++
 gnu/system/linux-initrd.scm | 52 +++++++++++++++++++++++++------------
 4 files changed, 85 insertions(+), 33 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index faa35060ef..9fd45ea209 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -15490,6 +15490,16 @@ operating-system Reference
 The list of Linux kernel modules that need to be available in the
 initial RAM disk.  @xref{Initial RAM Disk}.
 
+@item @code{initrd-extra-module-paths} (default: @code{'()})
+@cindex initrd
+@cindex initial RAM disk
+A list of paths outside of the Linux kernel tree to search for Linux
+kernel modules.
+
+The items in this may be either file-like objects (usually packages), or
+a list where the first element is a package and the second is an
+output--e.g. @code{(list (list zfs "module"))}.
+
 @item @code{initrd} (default: @code{base-initrd})
 A procedure that returns an initial RAM disk for the Linux
 kernel.  This field is provided to support low-level customization and
@@ -35816,6 +35826,19 @@ Initial RAM Disk
   (initrd-modules (cons "megaraid_sas" %base-initrd-modules)))
 @end lisp
 
+If a module listed in @code{initrd-modules} is not included in the
+Linux-libre kernel, then the location to it must be added to the
+@code{initrd-extra-module-paths} list.  For example, if your root file
+system exists on a ZFS pool, then your configuration might look like the
+following:
+
+@lisp
+(operating-system
+  ;; @dots{}
+  (initrd-modules (cons "zfs" %base-initrd-modules))
+  (initrd-extra-module-paths (list (list zfs "module"))))
+@end lisp
+
 @defvr {Scheme Variable} %base-initrd-modules
 This is the list of kernel modules included in the initrd by default.
 @end defvr
@@ -35929,13 +35952,15 @@ Initial RAM Disk
 @cindex initrd
 @cindex initial RAM disk
 @deffn {Scheme Procedure} raw-initrd @var{file-systems} @
-       [#:linux-modules '()] [#:mapped-devices '()] @
-       [#:keyboard-layout #f] @
+       [#:linux-modules '()] [#:linux-extra-module-paths '()] @
+       [#:mapped-devices '()] [#:keyboard-layout #f] @
        [#:helper-packages '()] [#:qemu-networking? #f] [#:volatile-root? #f]
 Return a derivation that builds a raw initrd.  @var{file-systems} is
 a list of file systems to be mounted by the initrd, possibly in addition to
 the root file system specified on the kernel command line via @option{root}.
 @var{linux-modules} is a list of kernel modules to be loaded at boot time.
+@var{linux-extra-module-paths} is a list of file-like objects to be searched
+for kernel modules.
 @var{mapped-devices} is a list of device mappings to realize before
 @var{file-systems} are mounted (@pxref{Mapped Devices}).
 @var{helper-packages} is a list of packages to be copied in the initrd.
@@ -35960,12 +35985,13 @@ Initial RAM Disk
 @deffn {Scheme Procedure} base-initrd @var{file-systems} @
        [#:mapped-devices '()] [#:keyboard-layout #f] @
        [#:qemu-networking? #f] [#:volatile-root? #f] @
-       [#:linux-modules '()]
-Return as a file-like object a generic initrd, with kernel
-modules taken from @var{linux}.  @var{file-systems} is a list of file-systems to be
-mounted by the initrd, possibly in addition to the root file system specified
-on the kernel command line via @option{root}.  @var{mapped-devices} is a list of device
-mappings to realize before @var{file-systems} are mounted.
+       [#:linux-modules '()] [#:linux-extra-module-paths '()]
+Return as a file-like object a generic initrd, with kernel modules taken
+from @var{linux} and @var{linux-extra-module-paths}.  @var{file-systems}
+is a list of file-systems to be mounted by the initrd, possibly in
+addition to the root file system specified on the kernel command line
+via @option{root}.  @var{mapped-devices} is a list of device mappings to
+realize before @var{file-systems} are mounted.
 
 When true, @var{keyboard-layout} is a @code{<keyboard-layout>} record denoting
 the desired console keyboard layout.  This is done before @var{mapped-devices}
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 053720574b..97b7e429ea 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -225,8 +225,8 @@ (define (file-name->module-name file)
 '.ko[.gz|.xz]' and normalizing it."
   (normalize-module-name (strip-extension (basename file))))
 
-(define (find-module-file directory module)
-  "Lookup module NAME under DIRECTORY, and return its absolute file name.
+(define (find-module-file directories module)
+  "Lookup module NAME under DIRECTORIES, and return its absolute file name.
 NAME can be a file name with or without '.ko', or it can be a module name.
 Raise an error if it could not be found.
 
@@ -247,16 +247,19 @@ (define (find-module-file directory module)
                            (else chr)))
                        module))))
 
-  (match (find-files directory
-                     (lambda (file stat)
-                       (member (strip-extension
-                                (basename file)) names)))
+  (match (append-map (lambda (directory)
+                       (find-files directory
+                                   (lambda (file _stat)
+                                     (member (strip-extension
+                                              (basename file))
+                                             names))))
+                       directories)
     ((file)
      file)
     (()
-     (error "kernel module not found" module directory))
+     (error "kernel module not found" module directories))
     ((_ ...)
-     (error "several modules by that name" module directory))))
+     (error "several modules by that name" module directories))))
 
 (define* (recursive-module-dependencies files
                                         #:key (lookup-module dot-ko))
diff --git a/gnu/system.scm b/gnu/system.scm
index ba3a1865d7..6c712e5cea 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -107,6 +107,7 @@ (define-module (gnu system)
             operating-system-label
             operating-system-default-label
             operating-system-initrd-modules
+            operating-system-initrd-extra-module-paths
             operating-system-initrd
             operating-system-users
             operating-system-groups
@@ -236,6 +237,8 @@ (define-record-type* <operating-system> operating-system
   (initrd-modules operating-system-initrd-modules ; list of strings
                   (thunked)                       ; it's system-dependent
                   (default %base-initrd-modules))
+  (initrd-extra-module-paths operating-system-initrd-extra-module-paths ; list of file-likes
+                             (default '()))
 
   (firmware operating-system-firmware             ; list of packages
             (default %base-firmware))
@@ -1312,6 +1315,8 @@ (define (operating-system-initrd-file os)
                #:linux (operating-system-kernel os)
                #:linux-modules
                (operating-system-initrd-modules os)
+               #:linux-extra-module-paths
+               (operating-system-initrd-extra-module-paths os)
                #:mapped-devices mapped-devices
                #:keyboard-layout (operating-system-keyboard-layout os)))
 
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 4c4c78e444..50a182d7d5 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -119,13 +119,23 @@ (define* (expression->initrd exp
                               `(#:references-graphs (("closure" ,init))))
                "/initrd.cpio.gz"))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory packages modules)
   "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+MODULES and taken from PACKAGES."
   (define imported-modules
     (source-module-closure '((gnu build linux-modules)
                              (guix build utils))))
 
+  (define (package+out->input package out)
+    (gexp-input package out))
+
+  (define package-inputs
+    (map (lambda (p)
+           (match p
+             ((p o) (package+out->input p o))
+             (p     (package+out->input p "out"))))
+         packages))
+
   (define build-exp
     (with-imported-modules imported-modules
       (with-extensions (list guile-zlib)
@@ -135,11 +145,12 @@ (define (flat-linux-module-directory linux modules)
                          (srfi srfi-1)
                          (srfi srfi-26))
 
-            (define module-dir
-              (string-append #$linux "/lib/modules"))
+            (define module-dirs
+              (map (cut string-append <> "/lib/modules")
+                   '#$package-inputs))
 
             (define modules
-              (let* ((lookup  (cut find-module-file module-dir <>))
+              (let* ((lookup  (cut find-module-file module-dirs <>))
                      (modules (map lookup '#$modules)))
                 (append modules
                         (recursive-module-dependencies
@@ -172,20 +183,23 @@ (define* (raw-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-paths '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       (helper-packages '())
                       qemu-networking?
                       volatile-root?
                       (on-error 'debug))
-  "Return as a file-like object a raw initrd, with kernel
-modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
-mounted by the initrd, possibly in addition to the root file system specified
-on the kernel command line via 'root'.  LINUX-MODULES is a list of kernel
-modules to be loaded at boot time. MAPPED-DEVICES is a list of device
-mappings to realize before FILE-SYSTEMS are mounted.
-HELPER-PACKAGES is a list of packages to be copied in the initrd. It may include
-e2fsck/static or other packages needed by the initrd to check root partition.
+  "Return as a file-like object a raw initrd, with kernel modules taken from
+LINUX.  FILE-SYSTEMS is a list of file-systems to be mounted by the initrd,
+possibly in addition to the root file system specified on the kernel command
+line via 'root'.  LINUX-MODULES is a list of kernel modules to be loaded at
+boot time. LINUX-EXTRA-MODULE-PATHS is a list of file-like objects which will
+be searched for modules in addition to the linux kernel. MAPPED-DEVICES is a
+list of device mappings to realize before FILE-SYSTEMS are mounted.
+HELPER-PACKAGES is a list of packages to be copied in the initrd. It may
+include e2fsck/static or other packages needed by the initrd to check root
+partition.
 
 When true, KEYBOARD-LAYOUT is a <keyboard-layout> record denoting the desired
 console keyboard layout.  This is done before MAPPED-DEVICES are set up and
@@ -221,7 +235,8 @@ (define* (raw-initrd file-systems
           #~())))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory (cons linux linux-extra-module-paths)
+                                 linux-modules))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -366,6 +381,7 @@ (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-paths '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       qemu-networking?
@@ -386,9 +402,10 @@ (define* (base-initrd file-systems
 QEMU-NETWORKING? and VOLATILE-ROOT? behaves as in raw-initrd.
 
 The initrd is automatically populated with all the kernel modules necessary
-for FILE-SYSTEMS and for the given options.  Additional kernel
-modules can be listed in LINUX-MODULES.  They will be added to the initrd, and
-loaded at boot time in the order in which they appear."
+for FILE-SYSTEMS and for the given options.  Additional kernel modules can be
+listed in LINUX-MODULES.  Additional search paths for modules can be listed in
+LINUX-EXTRA-MODULE-PATHS.  They will be added to the initrd, and loaded at
+boot time in the order in which they appear."
   (define linux-modules*
     ;; Modules added to the initrd and loaded from the initrd.
     `(,@linux-modules
@@ -408,6 +425,7 @@ (define* (base-initrd file-systems
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules*
+              #:linux-extra-module-paths linux-extra-module-paths
               #:mapped-devices mapped-devices
               #:helper-packages helper-packages
               #:keyboard-layout keyboard-layout
-- 
2.36.1





^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
                   ` (3 preceding siblings ...)
  2022-05-21 19:12 ` [bug#55231] [PATCH v2] " Brian Cully via Guix-patches via
@ 2022-05-30 20:14 ` Kaelyn via Guix-patches via
  2022-06-02 21:23 ` Kaelyn via Guix-patches via
                   ` (4 subsequent siblings)
  9 siblings, 0 replies; 23+ messages in thread
From: Kaelyn via Guix-patches via @ 2022-05-30 20:14 UTC (permalink / raw)
  To: 55231@debbugs.gnu.org


As the earlier patch discussion seemed to have focused on ZFS licensing issues that are largely orothogonal to the functionality being added, I'd like to again voice my support for the proposed patch. While ZFS is indeed a bad example due to open questions about module licensing and redistribution, it isn't the only out-of-tree module currently packaged in Guix which could be used with the extra initrd support (not counting future packages or external channels). Within the guix repo, the packages using the `linux-module-build-system` aside from zfs are:
- acpi-call-linux-module
- bbswitch-module
- ddcci-driver-linux
- rtl8812au-aircrack-ng-linux-module
- rtl8821ce-linux-module
- ttyebus-linux-module
- v4l2loopback-linux-module
- vhba-module
- wiregard-linux-compat

Of those nine, at least four look to be for specific hardware, and I can come up with theoretical use cases for why someone might want those or wiregard-linux-compat (though the cases are definitely contrived straw-men that may or may not be implementable in practice or actually useful in an initrd context, such as network booting with an older kernel and the root filesystem can only be accessed over a wireguard connection).

Cheers,
Kaelyn




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-21 19:12 ` [bug#55231] [PATCH v2] " Brian Cully via Guix-patches via
@ 2022-06-01 15:54   ` Ludovic Courtès
  2022-06-02 20:40     ` Brian Cully via Guix-patches via
  0 siblings, 1 reply; 23+ messages in thread
From: Ludovic Courtès @ 2022-06-01 15:54 UTC (permalink / raw)
  To: Brian Cully; +Cc: 55231, Brian Cully

Hello Brian,

Brian Cully <bjc@spork.org> skribis:

> From: Brian Cully <bjc@kublai.com>
>
> This patch allows copying of out-of-tree kernel modules to the Linux
> initrd.
>
> For out-of-tree modules to found, an extra slot has been added to
> ‘operating-system’: ‘initrd-extra-module-paths’, which contains a list
> containing items of either FILE-LIKE or (FILE-LIKE OUTPUT) which will be
> searched for modules in addition to the standard Linux module path. The
> required modules can then be added to ‘initrd-modules’ as normal and all paths
> will be searched for it, including for any modules depended on.
>
> * doc: Update documentation for ‘initrd-extra-module-paths’.
> * gnu/build/linux-modules.scm (find-module-file): change DIRECTORY argument to
> DIRECTORIES. Now takes a list of directories to search, rather than a single
> one.
> * gnu/system.scm <operating-system>: Add INITRD-EXTRA-MODULE-PATHS
> field and accessor. Takes a list of file-like objects.
> * gnu/system/linux-initrd.scm (flat-linux-module-directory): change LINUX
> argument to PACKAGES. Now contains a list of file-likes to search for modules.
> (raw-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
> to (flat-linux-extra-module-paths) along with the selected LINUX package.
> (base-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
> to (raw-initrd).

Nice, it looks like a welcome addition.

> +If a module listed in @code{initrd-modules} is not included in the
> +Linux-libre kernel, then the location to it must be added to the
> +@code{initrd-extra-module-paths} list.  For example, if your root file
> +system exists on a ZFS pool, then your configuration might look like the
> +following:
> +
> +@lisp
> +(operating-system
> +  ;; @dots{}
> +  (initrd-modules (cons "zfs" %base-initrd-modules))
> +  (initrd-extra-module-paths (list (list zfs "module"))))
> +@end lisp

I wonder if we could reuse the ‘kernel-loadable-modules’ field for this
purpose instead of introducing a new field.  We’d need to pass it to the
initrd procedures and have them search in there in addition to the
kernel package, pretty much like this patch already does actually.

WDYT?

Nitpick: the GNU convention is to use “path” to denote “search paths”,
and other “file”, “file name”, or similar.  In this case, that’d be
“kernel module” or “Linux module”.

Thank you!

Ludo’.




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-06-01 15:54   ` [bug#55231] [PATCH v1] " Ludovic Courtès
@ 2022-06-02 20:40     ` Brian Cully via Guix-patches via
  2022-06-03  7:27       ` Ludovic Courtès
  0 siblings, 1 reply; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-06-02 20:40 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 55231, Brian Cully


Ludovic Courtès <ludo@gnu.org> writes:

> I wonder if we could reuse the ‘kernel-loadable-modules’ field 
> for this
> purpose instead of introducing a new field.  We’d need to pass 
> it to the
> initrd procedures and have them search in there in addition to 
> the
> kernel package, pretty much like this patch already does 
> actually.

This sounds like it could be made to work as you suggest. My 
feeling is that the two contexts are slightly different, though, 
as the Linux modules are a superset of the initrd modules, so I’d 
prefer not to mix them as it might be confusing to people who are 
used to other Linux distros where the initrd modules are called 
out separately. I admit I’m probably being silly here, and don’t 
have any serious objection in principle.

> Nitpick: the GNU convention is to use “path” to denote “search 
> paths”,
> and other “file”, “file name”, or similar.  In this case, that’d 
> be
> “kernel module” or “Linux module”.

I struggled with this a fair amount, actually. What these 
file-likes actually represent is an element of a search path, even 
if they come in the odd form of a file-like object, which is why I 
used ‘path’. ‘file’ seems wrong, as it implies (to me) that it’s 
the ‘initrd-extra-module-files’ option itself that would include 
the module, rather than the ‘initrd-modules’ option.

Of course, all this goes away if we just reuse 
‘kernel-loadable-modules’ as an additional input, rather than 
adding another option, so that’s a distinct mark in favor of doing 
that.

When I get some time (hopefully soon!) I’ll try to thread 
‘kernel-loadable-modules’ through instead and see how far I can 
get with that approach. Do you think the documentation for it will 
need to be updated to specify that it’s also used as a search path 
for initrd building? Or maybe the better option is to update the 
documentation for ‘initrd-modules’ to say that it uses 
‘kernel-loadable-modules’ as input?

-bjc




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
                   ` (4 preceding siblings ...)
  2022-05-30 20:14 ` Kaelyn via Guix-patches via
@ 2022-06-02 21:23 ` Kaelyn via Guix-patches via
  2022-06-17  1:51 ` [bug#55231] [PATCH v2 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
                   ` (3 subsequent siblings)
  9 siblings, 0 replies; 23+ messages in thread
From: Kaelyn via Guix-patches via @ 2022-06-02 21:23 UTC (permalink / raw)
  To: 55231@debbugs.gnu.org

Hi,

My $0.02 on 'initrd-extra-module-paths' vs 'kernel-loadable-modules': if the initrd code is smart enough to not include the entire packages and only includes the requested modules from the packages (which I think is already true based on behavior observed some time ago), then not having to duplicate the list would be preferable.

I found this issue in the tracker after being surprised that the initrd builder ignored 'kernel-loadable-modules' with 'initrd-modules' only working for modules that are included in the primary linux kernel package. That surprise over the incompatibility between 'initrd-modules' and 'kernel-loadable-modules' is also why I've been relatively vocal about the patch. ;)

Cheers,
Kaelyn




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-06-02 20:40     ` Brian Cully via Guix-patches via
@ 2022-06-03  7:27       ` Ludovic Courtès
  0 siblings, 0 replies; 23+ messages in thread
From: Ludovic Courtès @ 2022-06-03  7:27 UTC (permalink / raw)
  To: Brian Cully; +Cc: 55231, Brian Cully

Hi,

Brian Cully <bjc@spork.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I wonder if we could reuse the ‘kernel-loadable-modules’ field for
>> this
>> purpose instead of introducing a new field.  We’d need to pass it to
>> the
>> initrd procedures and have them search in there in addition to the
>> kernel package, pretty much like this patch already does actually.
>
> This sounds like it could be made to work as you suggest. My feeling
> is that the two contexts are slightly different, though, as the Linux
> modules are a superset of the initrd modules,

Right.  Here, we want the initrd machinery to search for modules in any
place where they can be found, which is either ‘kernel’ or
‘kernel-loadable-modules’.

One could define ‘initrd-extra-module-paths’ to be different from
‘kernel-loadable-modules’, for example if you can be sure the module
will be loaded from the initrd and not after, but in general both are
likely to have the same value, no?

>> Nitpick: the GNU convention is to use “path” to denote “search
>> paths”,
>> and other “file”, “file name”, or similar.  In this case, that’d be
>> “kernel module” or “Linux module”.
>
> I struggled with this a fair amount, actually. What these file-likes
> actually represent is an element of a search path, even if they come
> in the odd form of a file-like object, which is why I used
> ‘path’. ‘file’ seems wrong, as it implies (to me) that it’s the
> ‘initrd-extra-module-files’ option itself that would include the
> module, rather than the ‘initrd-modules’ option.

Hmm right, you have a point!  :-)

> When I get some time (hopefully soon!) I’ll try to thread
> ‘kernel-loadable-modules’ through instead and see how far I can get
> with that approach. Do you think the documentation for it will need to
> be updated to specify that it’s also used as a search path for initrd
> building? Or maybe the better option is to update the documentation
> for ‘initrd-modules’ to say that it uses ‘kernel-loadable-modules’ as
> input?

I think you should update the documentation in the commit that changes
things, so that the patch is self-contained.

It may be enough to state in the documentation of the ‘initrd-modules’
field that its value is a list of module names that are searched for in
‘kernel’ and ‘kernel-loadable-modules’.

WDYT?

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v2 1/2] Allows copying of out-of-tree modules to the Linux initrd.
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
                   ` (5 preceding siblings ...)
  2022-06-02 21:23 ` Kaelyn via Guix-patches via
@ 2022-06-17  1:51 ` Brian Cully via Guix-patches via
  2022-06-17 20:34   ` [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Ludovic Courtès
  2022-06-18 19:11 ` [bug#55231] [PATCH v3 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
                   ` (2 subsequent siblings)
  9 siblings, 1 reply; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-06-17  1:51 UTC (permalink / raw)
  To: 55231; +Cc: Brian Cully

With this patch, modules for ‘initrd-modules’ will not only be searched for in
the in-tree Linux modules, but also any additional modules specified in
‘kernel-loadable-modules’.

* gnu/build/linux-modules.scm (find-module-file): change DIRECTORY argument to
DIRECTORIES. Now takes a list of directories to search, rather than a single
one.
* gnu/system/linux-initrd.scm (flat-linux-module-directory): change LINUX
argument to PACKAGES. Now contains a list of file-likes to search for modules.
(raw-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
to (flat-linux-extra-module-paths) along with the selected LINUX package.
(base-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
to (raw-initrd).
* gnu/system.scm (operating-system-initrd-file): pass in operating system
definition's kernel-loadable-modules into (make-initrd) as
LINUX-EXTRA-MODULE-PATHS.
---

I've removed the new operating-system slot in preference to re-using
kernel-loadable-modules, as discussed.

I did leave the old argument names for the lower level routines in place,
as I feel that they more accurately represent how the data are being
used.

I've also pulled out the documentation. Once this patch set is beaten
into acceptability, the documentation can be adjusted if that's deemed
necessary.

 gnu/build/linux-modules.scm | 19 ++++++++------
 gnu/system.scm              |  2 ++
 gnu/system/linux-initrd.scm | 52 +++++++++++++++++++++++++------------
 3 files changed, 48 insertions(+), 25 deletions(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 053720574b..97b7e429ea 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -225,8 +225,8 @@ (define (file-name->module-name file)
 '.ko[.gz|.xz]' and normalizing it."
   (normalize-module-name (strip-extension (basename file))))
 
-(define (find-module-file directory module)
-  "Lookup module NAME under DIRECTORY, and return its absolute file name.
+(define (find-module-file directories module)
+  "Lookup module NAME under DIRECTORIES, and return its absolute file name.
 NAME can be a file name with or without '.ko', or it can be a module name.
 Raise an error if it could not be found.
 
@@ -247,16 +247,19 @@ (define (find-module-file directory module)
                            (else chr)))
                        module))))
 
-  (match (find-files directory
-                     (lambda (file stat)
-                       (member (strip-extension
-                                (basename file)) names)))
+  (match (append-map (lambda (directory)
+                       (find-files directory
+                                   (lambda (file _stat)
+                                     (member (strip-extension
+                                              (basename file))
+                                             names))))
+                       directories)
     ((file)
      file)
     (()
-     (error "kernel module not found" module directory))
+     (error "kernel module not found" module directories))
     ((_ ...)
-     (error "several modules by that name" module directory))))
+     (error "several modules by that name" module directories))))
 
 (define* (recursive-module-dependencies files
                                         #:key (lookup-module dot-ko))
diff --git a/gnu/system.scm b/gnu/system.scm
index ba1b7b5152..6f52377c8d 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -1313,6 +1313,8 @@ (define (operating-system-initrd-file os)
                #:linux (operating-system-kernel os)
                #:linux-modules
                (operating-system-initrd-modules os)
+               #:linux-extra-module-paths
+               (operating-system-kernel-loadable-modules os)
                #:mapped-devices mapped-devices
                #:keyboard-layout (operating-system-keyboard-layout os)))
 
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 4c4c78e444..50a182d7d5 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -119,13 +119,23 @@ (define* (expression->initrd exp
                               `(#:references-graphs (("closure" ,init))))
                "/initrd.cpio.gz"))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory packages modules)
   "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+MODULES and taken from PACKAGES."
   (define imported-modules
     (source-module-closure '((gnu build linux-modules)
                              (guix build utils))))
 
+  (define (package+out->input package out)
+    (gexp-input package out))
+
+  (define package-inputs
+    (map (lambda (p)
+           (match p
+             ((p o) (package+out->input p o))
+             (p     (package+out->input p "out"))))
+         packages))
+
   (define build-exp
     (with-imported-modules imported-modules
       (with-extensions (list guile-zlib)
@@ -135,11 +145,12 @@ (define (flat-linux-module-directory linux modules)
                          (srfi srfi-1)
                          (srfi srfi-26))
 
-            (define module-dir
-              (string-append #$linux "/lib/modules"))
+            (define module-dirs
+              (map (cut string-append <> "/lib/modules")
+                   '#$package-inputs))
 
             (define modules
-              (let* ((lookup  (cut find-module-file module-dir <>))
+              (let* ((lookup  (cut find-module-file module-dirs <>))
                      (modules (map lookup '#$modules)))
                 (append modules
                         (recursive-module-dependencies
@@ -172,20 +183,23 @@ (define* (raw-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-paths '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       (helper-packages '())
                       qemu-networking?
                       volatile-root?
                       (on-error 'debug))
-  "Return as a file-like object a raw initrd, with kernel
-modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
-mounted by the initrd, possibly in addition to the root file system specified
-on the kernel command line via 'root'.  LINUX-MODULES is a list of kernel
-modules to be loaded at boot time. MAPPED-DEVICES is a list of device
-mappings to realize before FILE-SYSTEMS are mounted.
-HELPER-PACKAGES is a list of packages to be copied in the initrd. It may include
-e2fsck/static or other packages needed by the initrd to check root partition.
+  "Return as a file-like object a raw initrd, with kernel modules taken from
+LINUX.  FILE-SYSTEMS is a list of file-systems to be mounted by the initrd,
+possibly in addition to the root file system specified on the kernel command
+line via 'root'.  LINUX-MODULES is a list of kernel modules to be loaded at
+boot time. LINUX-EXTRA-MODULE-PATHS is a list of file-like objects which will
+be searched for modules in addition to the linux kernel. MAPPED-DEVICES is a
+list of device mappings to realize before FILE-SYSTEMS are mounted.
+HELPER-PACKAGES is a list of packages to be copied in the initrd. It may
+include e2fsck/static or other packages needed by the initrd to check root
+partition.
 
 When true, KEYBOARD-LAYOUT is a <keyboard-layout> record denoting the desired
 console keyboard layout.  This is done before MAPPED-DEVICES are set up and
@@ -221,7 +235,8 @@ (define* (raw-initrd file-systems
           #~())))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory (cons linux linux-extra-module-paths)
+                                 linux-modules))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -366,6 +381,7 @@ (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-paths '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       qemu-networking?
@@ -386,9 +402,10 @@ (define* (base-initrd file-systems
 QEMU-NETWORKING? and VOLATILE-ROOT? behaves as in raw-initrd.
 
 The initrd is automatically populated with all the kernel modules necessary
-for FILE-SYSTEMS and for the given options.  Additional kernel
-modules can be listed in LINUX-MODULES.  They will be added to the initrd, and
-loaded at boot time in the order in which they appear."
+for FILE-SYSTEMS and for the given options.  Additional kernel modules can be
+listed in LINUX-MODULES.  Additional search paths for modules can be listed in
+LINUX-EXTRA-MODULE-PATHS.  They will be added to the initrd, and loaded at
+boot time in the order in which they appear."
   (define linux-modules*
     ;; Modules added to the initrd and loaded from the initrd.
     `(,@linux-modules
@@ -408,6 +425,7 @@ (define* (base-initrd file-systems
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules*
+              #:linux-extra-module-paths linux-extra-module-paths
               #:mapped-devices mapped-devices
               #:helper-packages helper-packages
               #:keyboard-layout keyboard-layout
-- 
2.36.1





^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-06-17  1:51 ` [bug#55231] [PATCH v2 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
@ 2022-06-17 20:34   ` Ludovic Courtès
  0 siblings, 0 replies; 23+ messages in thread
From: Ludovic Courtès @ 2022-06-17 20:34 UTC (permalink / raw)
  To: Brian Cully; +Cc: 55231

Hi Brian,

Brian Cully <bjc@spork.org> skribis:

> With this patch, modules for ‘initrd-modules’ will not only be searched for in
> the in-tree Linux modules, but also any additional modules specified in
> ‘kernel-loadable-modules’.
>
> * gnu/build/linux-modules.scm (find-module-file): change DIRECTORY argument to
> DIRECTORIES. Now takes a list of directories to search, rather than a single
> one.
> * gnu/system/linux-initrd.scm (flat-linux-module-directory): change LINUX
> argument to PACKAGES. Now contains a list of file-likes to search for modules.
> (raw-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
> to (flat-linux-extra-module-paths) along with the selected LINUX package.
> (base-initrd): Add LINUX-EXTRA-MODULE-PATHS keyword argument. Pass it
> to (raw-initrd).
> * gnu/system.scm (operating-system-initrd-file): pass in operating system
> definition's kernel-loadable-modules into (make-initrd) as
> LINUX-EXTRA-MODULE-PATHS.
> ---
>
> I've removed the new operating-system slot in preference to re-using
> kernel-loadable-modules, as discussed.
>
> I did leave the old argument names for the lower level routines in place,
> as I feel that they more accurately represent how the data are being
> used.
>
> I've also pulled out the documentation. Once this patch set is beaten
> into acceptability, the documentation can be adjusted if that's deemed
> necessary.

Alright!

It looks great to me.  I have two cosmetic comments:

> +  (define (package+out->input package out)
> +    (gexp-input package out))

I think you can remove this definition and use ‘gexp-input’ directly.

> @@ -172,20 +183,23 @@ (define* (raw-initrd file-systems
>                        #:key
>                        (linux linux-libre)
>                        (linux-modules '())
> +                      (linux-extra-module-paths '())

Nitpick: I’d call it #:linux-extra-module-path (singular, as in “search
path”) or simply #:linux-extra-module-directories.

If you could introduce a couple of lines in doc/guix.texi to explain
where modules are searched for, that’d be perfect.

Thanks for taking the time to prepare this revision!

Ludo’.




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v3 1/2] Allows copying of out-of-tree modules to the Linux initrd.
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
                   ` (6 preceding siblings ...)
  2022-06-17  1:51 ` [bug#55231] [PATCH v2 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
@ 2022-06-18 19:11 ` Brian Cully via Guix-patches via
  2022-06-18 19:11   ` [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’ Brian Cully via Guix-patches via
  2022-06-21 12:34 ` [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Kaelyn via Guix-patches via
  2022-06-24 13:28 ` [bug#55231] [PATCH v4 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
  9 siblings, 1 reply; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-06-18 19:11 UTC (permalink / raw)
  To: 55231; +Cc: Brian Cully

With this patch, modules for ‘initrd-modules’ will not only be searched for in
the in-tree Linux modules, but also any additional modules specified in
‘kernel-loadable-modules’.

* gnu/build/linux-modules.scm (find-module-file): change DIRECTORY argument to
DIRECTORIES. Now takes a list of directories to search, rather than a single
one.
* gnu/system/linux-initrd.scm (flat-linux-module-directory): change LINUX
argument to PACKAGES. Now contains a list of file-likes to search for modules.
(raw-initrd): Add LINUX-EXTRA-MODULE-DIRECTORIES keyword argument. Pass it
to (flat-linux-module-directory) along with the selected LINUX package.
(base-initrd): Add LINUX-EXTRA-MODULE-DIRECTORIES keyword argument. Pass it
to (raw-initrd).
* gnu/system.scm (operating-system-initrd-file): pass in operating system
definition's kernel-loadable-modules into (make-initrd) as
LINUX-EXTRA-MODULE-DIRECTORIES.
---

This version removes the redundant (package+out->input) procedure, and
renames ‘linux-extra-module-paths’ to ‘linux-extra-module-directories’.

 gnu/build/linux-modules.scm | 19 ++++++++------
 gnu/system.scm              |  2 ++
 gnu/system/linux-initrd.scm | 49 ++++++++++++++++++++++++-------------
 3 files changed, 45 insertions(+), 25 deletions(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 053720574b..97b7e429ea 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -225,8 +225,8 @@ (define (file-name->module-name file)
 '.ko[.gz|.xz]' and normalizing it."
   (normalize-module-name (strip-extension (basename file))))
 
-(define (find-module-file directory module)
-  "Lookup module NAME under DIRECTORY, and return its absolute file name.
+(define (find-module-file directories module)
+  "Lookup module NAME under DIRECTORIES, and return its absolute file name.
 NAME can be a file name with or without '.ko', or it can be a module name.
 Raise an error if it could not be found.
 
@@ -247,16 +247,19 @@ (define (find-module-file directory module)
                            (else chr)))
                        module))))
 
-  (match (find-files directory
-                     (lambda (file stat)
-                       (member (strip-extension
-                                (basename file)) names)))
+  (match (append-map (lambda (directory)
+                       (find-files directory
+                                   (lambda (file _stat)
+                                     (member (strip-extension
+                                              (basename file))
+                                             names))))
+                       directories)
     ((file)
      file)
     (()
-     (error "kernel module not found" module directory))
+     (error "kernel module not found" module directories))
     ((_ ...)
-     (error "several modules by that name" module directory))))
+     (error "several modules by that name" module directories))))
 
 (define* (recursive-module-dependencies files
                                         #:key (lookup-module dot-ko))
diff --git a/gnu/system.scm b/gnu/system.scm
index ba1b7b5152..2439560671 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -1313,6 +1313,8 @@ (define (operating-system-initrd-file os)
                #:linux (operating-system-kernel os)
                #:linux-modules
                (operating-system-initrd-modules os)
+               #:linux-extra-module-directories
+               (operating-system-kernel-loadable-modules os)
                #:mapped-devices mapped-devices
                #:keyboard-layout (operating-system-keyboard-layout os)))
 
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 4c4c78e444..f6e8f75efa 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -119,13 +119,20 @@ (define* (expression->initrd exp
                               `(#:references-graphs (("closure" ,init))))
                "/initrd.cpio.gz"))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory packages modules)
   "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+MODULES and taken from PACKAGES."
   (define imported-modules
     (source-module-closure '((gnu build linux-modules)
                              (guix build utils))))
 
+  (define package-inputs
+    (map (lambda (p)
+           (match p
+             ((p o) (gexp-input p o))
+             (p     (gexp-input p "out"))))
+         packages))
+
   (define build-exp
     (with-imported-modules imported-modules
       (with-extensions (list guile-zlib)
@@ -135,11 +142,12 @@ (define (flat-linux-module-directory linux modules)
                          (srfi srfi-1)
                          (srfi srfi-26))
 
-            (define module-dir
-              (string-append #$linux "/lib/modules"))
+            (define module-dirs
+              (map (cut string-append <> "/lib/modules")
+                   '#$package-inputs))
 
             (define modules
-              (let* ((lookup  (cut find-module-file module-dir <>))
+              (let* ((lookup  (cut find-module-file module-dirs <>))
                      (modules (map lookup '#$modules)))
                 (append modules
                         (recursive-module-dependencies
@@ -172,20 +180,23 @@ (define* (raw-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-directories '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       (helper-packages '())
                       qemu-networking?
                       volatile-root?
                       (on-error 'debug))
-  "Return as a file-like object a raw initrd, with kernel
-modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
-mounted by the initrd, possibly in addition to the root file system specified
-on the kernel command line via 'root'.  LINUX-MODULES is a list of kernel
-modules to be loaded at boot time. MAPPED-DEVICES is a list of device
-mappings to realize before FILE-SYSTEMS are mounted.
-HELPER-PACKAGES is a list of packages to be copied in the initrd. It may include
-e2fsck/static or other packages needed by the initrd to check root partition.
+  "Return as a file-like object a raw initrd, with kernel modules taken from
+LINUX.  FILE-SYSTEMS is a list of file-systems to be mounted by the initrd,
+possibly in addition to the root file system specified on the kernel command
+line via 'root'.  LINUX-MODULES is a list of kernel modules to be loaded at
+boot time. LINUX-EXTRA-MODULE-DIRECTORIES is a list of file-like objects which
+will be searched for modules in addition to the linux kernel. MAPPED-DEVICES
+is a list of device mappings to realize before FILE-SYSTEMS are mounted.
+HELPER-PACKAGES is a list of packages to be copied in the initrd. It may
+include e2fsck/static or other packages needed by the initrd to check root
+partition.
 
 When true, KEYBOARD-LAYOUT is a <keyboard-layout> record denoting the desired
 console keyboard layout.  This is done before MAPPED-DEVICES are set up and
@@ -221,7 +232,8 @@ (define* (raw-initrd file-systems
           #~())))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory (cons linux linux-extra-module-directories)
+                                 linux-modules))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -366,6 +378,7 @@ (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-directories '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       qemu-networking?
@@ -386,9 +399,10 @@ (define* (base-initrd file-systems
 QEMU-NETWORKING? and VOLATILE-ROOT? behaves as in raw-initrd.
 
 The initrd is automatically populated with all the kernel modules necessary
-for FILE-SYSTEMS and for the given options.  Additional kernel
-modules can be listed in LINUX-MODULES.  They will be added to the initrd, and
-loaded at boot time in the order in which they appear."
+for FILE-SYSTEMS and for the given options.  Additional kernel modules can be
+listed in LINUX-MODULES.  Additional directories for modules can be listed in
+LINUX-EXTRA-MODULE-DIRECTORIES.  They will be added to the initrd, and loaded
+at boot time in the order in which they appear."
   (define linux-modules*
     ;; Modules added to the initrd and loaded from the initrd.
     `(,@linux-modules
@@ -408,6 +422,7 @@ (define* (base-initrd file-systems
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules*
+              #:linux-extra-module-directories linux-extra-module-directories
               #:mapped-devices mapped-devices
               #:helper-packages helper-packages
               #:keyboard-layout keyboard-layout
-- 
2.36.1





^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’.
  2022-06-18 19:11 ` [bug#55231] [PATCH v3 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
@ 2022-06-18 19:11   ` Brian Cully via Guix-patches via
  2022-06-18 20:34     ` Maxime Devos
  0 siblings, 1 reply; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-06-18 19:11 UTC (permalink / raw)
  To: 55231; +Cc: Brian Cully

---
 doc/guix.texi | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index eda0956260..d97909695f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -36461,6 +36461,19 @@ Initial RAM Disk
   (initrd-modules (cons "megaraid_sas" %base-initrd-modules)))
 @end lisp
 
+If a module listed in @code{initrd-modules} is not included in the
+Linux-libre kernel, then the location to it must be added to the
+@code{kernel-loadable-modules} list.  For example, if your root file
+system exists on a ZFS pool, then your configuration might look like the
+following:
+
+@lisp
+(operating-system
+  ;; @dots{}
+  (initrd-modules (cons "zfs" %base-initrd-modules))
+  (kernel-loadable-modules (list (list zfs "module"))))
+@end lisp
+
 @defvr {Scheme Variable} %base-initrd-modules
 This is the list of kernel modules included in the initrd by default.
 @end defvr
-- 
2.36.1





^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’.
  2022-06-18 19:11   ` [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’ Brian Cully via Guix-patches via
@ 2022-06-18 20:34     ` Maxime Devos
  2022-06-18 20:43       ` Brian Cully via Guix-patches via
  0 siblings, 1 reply; 23+ messages in thread
From: Maxime Devos @ 2022-06-18 20:34 UTC (permalink / raw)
  To: Brian Cully, 55231

[-- Attachment #1: Type: text/plain, Size: 1487 bytes --]

Brian Cully via Guix-patches via schreef op za 18-06-2022 om 15:11 [-
0400]:
> +If a module listed in @code{initrd-modules} is not included in the
> +Linux-libre kernel, then the location to it must be added to the
> +@code{kernel-loadable-modules} list.  For example, if your root file
> +system exists on a ZFS pool, then your configuration might look like the
> +following:
> +
> +@lisp
> +(operating-system
> +  ;; @dots{}
> +  (initrd-modules (cons "zfs" %base-initrd-modules))
> +  (kernel-loadable-modules (list (list zfs "module"))))
> +@end lisp

As written previously, this is not a good example, because:

> As-is, I don't think this is a good example, because
> 'expression->initrd' does not set #:substitutable? #false,
> the 'zfs' package has the comment (*)
> 
>      `(;; The ZFS kernel module should not be downloaded since the
> license
>        ;; terms don't allow for distributing it, only building it
> locally.
>        #:substitutable? #f [...])
> 
> and IIUC, the code inside expression->initrd copies the kernel module
> into the new store item, so it looks like this accidentally suggest
> people to commit copyvios, and copyvios are currently against the
> law.

and because the defense for not considering the ZFS license to be a
problem consists of the ZFS module not being distributed in binary
form, whereas this suggestion would in some situations cause it to be
distributed in binary form.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’.
  2022-06-18 20:34     ` Maxime Devos
@ 2022-06-18 20:43       ` Brian Cully via Guix-patches via
  2022-06-18 22:34         ` Maxime Devos
  0 siblings, 1 reply; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-06-18 20:43 UTC (permalink / raw)
  To: Maxime Devos; +Cc: 55231


> and because the defense for not considering the ZFS license to 
> be a
> problem consists of the ZFS module not being distributed in 
> binary
> form, whereas this suggestion would in some situations cause it 
> to be
> distributed in binary form.

If you have another example, I'll put it in instead. I used this 
one, because I know this one works, and that's all.

That said, I'm not sure how this would cause the module to be 
distributed as a binary. In order for it to be added to the 
initrd, it will still need to built using the DKMS mechanism, and 
thus compiled on (or at least for) the target Guix installation.

If the complaint is that one could generate a USB stick, or some 
such, with ZFS in the initrd, then yes, that's possible. But 
that's also possible by using the existing 
‘kernel-loadable-modules’ mechanism to generate an image with 
‘guix system image’ and distributing that. I don't think it's our 
job to try and prevent such things, since, even if desirable, it's 
not really feasible.

If the complaint is merely that it's in the documentation, then 
ok, I'll change it to whatever module you want.

-bjc




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’.
  2022-06-18 20:43       ` Brian Cully via Guix-patches via
@ 2022-06-18 22:34         ` Maxime Devos
  2022-06-18 23:11           ` Brian Cully via Guix-patches via
  0 siblings, 1 reply; 23+ messages in thread
From: Maxime Devos @ 2022-06-18 22:34 UTC (permalink / raw)
  To: Brian Cully; +Cc: 55231

[-- Attachment #1: Type: text/plain, Size: 483 bytes --]

Brian Cully schreef op za 18-06-2022 om 16:43 [-0400]:
> That said, I'm not sure how this would cause the module to be 
> distributed as a binary.  [...]

Because #:substitutable? isn't set appropriately in dependents,
substitute servers exist, ZFS can be used on substitute servers and
someone using the substitute server might have a sufficiently similar
system configuration to do substitution of a store item containing a
copy of the ZFS binary.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’.
  2022-06-18 22:34         ` Maxime Devos
@ 2022-06-18 23:11           ` Brian Cully via Guix-patches via
  2022-06-19 12:05             ` Maxime Devos
  0 siblings, 1 reply; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-06-18 23:11 UTC (permalink / raw)
  To: Maxime Devos; +Cc: 55231


Maxime Devos <maximedevos@telenet.be> writes:

> Because #:substitutable? isn't set appropriately in dependents,
> substitute servers exist, ZFS can be used on substitute servers 
> and
> someone using the substitute server might have a sufficiently 
> similar
> system configuration to do substitution of a store item 
> containing a
> copy of the ZFS binary.

So, simply using ZFS in ‘kernel-loadable-modules’ would be enough 
to trigger this misbehavior? That sounds like a pretty serious 
issue. Would it be possible to have the substitute servers filter 
on the #:substitutable flag?

This is getting out of scope, though. I don't really want this 
patch to go in without /some/ documentation regarding how modules 
get looked up, so if anyone has an alternate module and use-case, 
I'll just swap it in for the ZFS one.

-bjc




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’.
  2022-06-18 23:11           ` Brian Cully via Guix-patches via
@ 2022-06-19 12:05             ` Maxime Devos
  0 siblings, 0 replies; 23+ messages in thread
From: Maxime Devos @ 2022-06-19 12:05 UTC (permalink / raw)
  To: Brian Cully; +Cc: 55231

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

Brian Cully schreef op za 18-06-2022 om 19:11 [-0400]:
> So, simply using ZFS in ‘kernel-loadable-modules’ would be enough 
> to trigger this misbehavior? That sounds like a pretty serious 
> issue. Would it be possible to have the substitute servers filter 
> on the #:substitutable flag?

That's exactly what the #:substitutable flag is for (IIUC)!
However, the problem is that currently, that flag isn't set for
derivations that make a copy of the ZFS module.  As I wrote previously:

> > and IIUC, the code inside expression->initrd copies the kernel
> > module into the new store item,

> This is getting out of scope, though. I don't really want this 
> patch to go in without /some/ documentation regarding how modules 
> get looked up, so if anyone has an alternate module and use-case, 
> I'll just swap it in for the ZFS one.

I don't have any.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
                   ` (7 preceding siblings ...)
  2022-06-18 19:11 ` [bug#55231] [PATCH v3 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
@ 2022-06-21 12:34 ` Kaelyn via Guix-patches via
  2022-06-24 13:28 ` [bug#55231] [PATCH v4 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
  9 siblings, 0 replies; 23+ messages in thread
From: Kaelyn via Guix-patches via @ 2022-06-21 12:34 UTC (permalink / raw)
  To: 55231@debbugs.gnu.org




Hi again,

Regarding the documentation example, one (contrived) alternative is:

"""
For example, if you need the driver for a Realtek RTL8821CE wireless network adapter for mounting the root filesystem over NFS, your configuration might include the following:

@lisp
(operating-system
  ;; @dots{}
  (initrd-modules (cons "8821ce" %base-initrd-modules))
  (kernel-loadable-modules (list (list rtl8821ce-linux-module "module"))))
@end lisp
"""

While I don't have the hardware, I did verify the kernel module name by building the rtl8821ce-linux-module package.

Cheers,
Kaelyn




^ permalink raw reply	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v4 1/2] Allows copying of out-of-tree modules to the Linux initrd.
  2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
                   ` (8 preceding siblings ...)
  2022-06-21 12:34 ` [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Kaelyn via Guix-patches via
@ 2022-06-24 13:28 ` Brian Cully via Guix-patches via
  2022-06-24 13:28   ` [bug#55231] [PATCH v4 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’ Brian Cully via Guix-patches via
  9 siblings, 1 reply; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-06-24 13:28 UTC (permalink / raw)
  To: 55231; +Cc: Brian Cully

With this patch, modules for ‘initrd-modules’ will not only be searched for in
the in-tree Linux modules, but also any additional modules specified in
‘kernel-loadable-modules’.

* gnu/build/linux-modules.scm (find-module-file): change DIRECTORY argument to
DIRECTORIES. Now takes a list of directories to search, rather than a single
one.
* gnu/system/linux-initrd.scm (flat-linux-module-directory): change LINUX
argument to PACKAGES. Now contains a list of file-likes to search for modules.
(raw-initrd): Add LINUX-EXTRA-MODULE-DIRECTORIES keyword argument. Pass it
to (flat-linux-module-directory) along with the selected LINUX package.
(base-initrd): Add LINUX-EXTRA-MODULE-DIRECTORIES keyword argument. Pass it
to (raw-initrd).
* gnu/system.scm (operating-system-initrd-file): pass in operating system
definition's kernel-loadable-modules into (make-initrd) as
LINUX-EXTRA-MODULE-DIRECTORIES.
---
 gnu/build/linux-modules.scm | 19 ++++++++------
 gnu/system.scm              |  2 ++
 gnu/system/linux-initrd.scm | 49 ++++++++++++++++++++++++-------------
 3 files changed, 45 insertions(+), 25 deletions(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 053720574b..97b7e429ea 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -225,8 +225,8 @@ (define (file-name->module-name file)
 '.ko[.gz|.xz]' and normalizing it."
   (normalize-module-name (strip-extension (basename file))))
 
-(define (find-module-file directory module)
-  "Lookup module NAME under DIRECTORY, and return its absolute file name.
+(define (find-module-file directories module)
+  "Lookup module NAME under DIRECTORIES, and return its absolute file name.
 NAME can be a file name with or without '.ko', or it can be a module name.
 Raise an error if it could not be found.
 
@@ -247,16 +247,19 @@ (define (find-module-file directory module)
                            (else chr)))
                        module))))
 
-  (match (find-files directory
-                     (lambda (file stat)
-                       (member (strip-extension
-                                (basename file)) names)))
+  (match (append-map (lambda (directory)
+                       (find-files directory
+                                   (lambda (file _stat)
+                                     (member (strip-extension
+                                              (basename file))
+                                             names))))
+                       directories)
     ((file)
      file)
     (()
-     (error "kernel module not found" module directory))
+     (error "kernel module not found" module directories))
     ((_ ...)
-     (error "several modules by that name" module directory))))
+     (error "several modules by that name" module directories))))
 
 (define* (recursive-module-dependencies files
                                         #:key (lookup-module dot-ko))
diff --git a/gnu/system.scm b/gnu/system.scm
index ba1b7b5152..2439560671 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -1313,6 +1313,8 @@ (define (operating-system-initrd-file os)
                #:linux (operating-system-kernel os)
                #:linux-modules
                (operating-system-initrd-modules os)
+               #:linux-extra-module-directories
+               (operating-system-kernel-loadable-modules os)
                #:mapped-devices mapped-devices
                #:keyboard-layout (operating-system-keyboard-layout os)))
 
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 4c4c78e444..f6e8f75efa 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -119,13 +119,20 @@ (define* (expression->initrd exp
                               `(#:references-graphs (("closure" ,init))))
                "/initrd.cpio.gz"))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory packages modules)
   "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+MODULES and taken from PACKAGES."
   (define imported-modules
     (source-module-closure '((gnu build linux-modules)
                              (guix build utils))))
 
+  (define package-inputs
+    (map (lambda (p)
+           (match p
+             ((p o) (gexp-input p o))
+             (p     (gexp-input p "out"))))
+         packages))
+
   (define build-exp
     (with-imported-modules imported-modules
       (with-extensions (list guile-zlib)
@@ -135,11 +142,12 @@ (define (flat-linux-module-directory linux modules)
                          (srfi srfi-1)
                          (srfi srfi-26))
 
-            (define module-dir
-              (string-append #$linux "/lib/modules"))
+            (define module-dirs
+              (map (cut string-append <> "/lib/modules")
+                   '#$package-inputs))
 
             (define modules
-              (let* ((lookup  (cut find-module-file module-dir <>))
+              (let* ((lookup  (cut find-module-file module-dirs <>))
                      (modules (map lookup '#$modules)))
                 (append modules
                         (recursive-module-dependencies
@@ -172,20 +180,23 @@ (define* (raw-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-directories '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       (helper-packages '())
                       qemu-networking?
                       volatile-root?
                       (on-error 'debug))
-  "Return as a file-like object a raw initrd, with kernel
-modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
-mounted by the initrd, possibly in addition to the root file system specified
-on the kernel command line via 'root'.  LINUX-MODULES is a list of kernel
-modules to be loaded at boot time. MAPPED-DEVICES is a list of device
-mappings to realize before FILE-SYSTEMS are mounted.
-HELPER-PACKAGES is a list of packages to be copied in the initrd. It may include
-e2fsck/static or other packages needed by the initrd to check root partition.
+  "Return as a file-like object a raw initrd, with kernel modules taken from
+LINUX.  FILE-SYSTEMS is a list of file-systems to be mounted by the initrd,
+possibly in addition to the root file system specified on the kernel command
+line via 'root'.  LINUX-MODULES is a list of kernel modules to be loaded at
+boot time. LINUX-EXTRA-MODULE-DIRECTORIES is a list of file-like objects which
+will be searched for modules in addition to the linux kernel. MAPPED-DEVICES
+is a list of device mappings to realize before FILE-SYSTEMS are mounted.
+HELPER-PACKAGES is a list of packages to be copied in the initrd. It may
+include e2fsck/static or other packages needed by the initrd to check root
+partition.
 
 When true, KEYBOARD-LAYOUT is a <keyboard-layout> record denoting the desired
 console keyboard layout.  This is done before MAPPED-DEVICES are set up and
@@ -221,7 +232,8 @@ (define* (raw-initrd file-systems
           #~())))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory (cons linux linux-extra-module-directories)
+                                 linux-modules))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -366,6 +378,7 @@ (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (linux-extra-module-directories '())
                       (mapped-devices '())
                       (keyboard-layout #f)
                       qemu-networking?
@@ -386,9 +399,10 @@ (define* (base-initrd file-systems
 QEMU-NETWORKING? and VOLATILE-ROOT? behaves as in raw-initrd.
 
 The initrd is automatically populated with all the kernel modules necessary
-for FILE-SYSTEMS and for the given options.  Additional kernel
-modules can be listed in LINUX-MODULES.  They will be added to the initrd, and
-loaded at boot time in the order in which they appear."
+for FILE-SYSTEMS and for the given options.  Additional kernel modules can be
+listed in LINUX-MODULES.  Additional directories for modules can be listed in
+LINUX-EXTRA-MODULE-DIRECTORIES.  They will be added to the initrd, and loaded
+at boot time in the order in which they appear."
   (define linux-modules*
     ;; Modules added to the initrd and loaded from the initrd.
     `(,@linux-modules
@@ -408,6 +422,7 @@ (define* (base-initrd file-systems
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules*
+              #:linux-extra-module-directories linux-extra-module-directories
               #:mapped-devices mapped-devices
               #:helper-packages helper-packages
               #:keyboard-layout keyboard-layout
-- 
2.36.1





^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [bug#55231] [PATCH v4 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’.
  2022-06-24 13:28 ` [bug#55231] [PATCH v4 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
@ 2022-06-24 13:28   ` Brian Cully via Guix-patches via
  0 siblings, 0 replies; 23+ messages in thread
From: Brian Cully via Guix-patches via @ 2022-06-24 13:28 UTC (permalink / raw)
  To: 55231; +Cc: Brian Cully

---
 doc/guix.texi | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index eda0956260..7c4682a76d 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -36461,6 +36461,21 @@ Initial RAM Disk
   (initrd-modules (cons "megaraid_sas" %base-initrd-modules)))
 @end lisp
 
+If a module listed in @code{initrd-modules} is not included in the
+Linux-libre kernel, then the location to it must be added to the
+@code{kernel-loadable-modules} list.
+
+For example, if you need the driver for a Realtek RTL8821CE wireless
+network adapter for mounting the root filesystem over NFS, your
+configuration might include the following:
+
+@lisp
+(operating-system
+  ;; @dots{}
+  (initrd-modules (cons "8821ce" %base-initrd-modules))
+  (kernel-loadable-modules (list (list rtl8821ce-linux-module "module"))))
+@end lisp
+
 @defvr {Scheme Variable} %base-initrd-modules
 This is the list of kernel modules included in the initrd by default.
 @end defvr
-- 
2.36.1





^ permalink raw reply related	[flat|nested] 23+ messages in thread

end of thread, other threads:[~2022-06-24 13:30 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-02 19:11 [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Brian Cully via Guix-patches via
2022-05-02 22:03 ` Maxime Devos
2022-05-02 22:53   ` Brian Cully via Guix-patches via
2022-05-13 19:46 ` Kaelyn via Guix-patches via
2022-05-13 20:25 ` Maxime Devos
2022-05-21 19:12 ` [bug#55231] [PATCH v2] " Brian Cully via Guix-patches via
2022-06-01 15:54   ` [bug#55231] [PATCH v1] " Ludovic Courtès
2022-06-02 20:40     ` Brian Cully via Guix-patches via
2022-06-03  7:27       ` Ludovic Courtès
2022-05-30 20:14 ` Kaelyn via Guix-patches via
2022-06-02 21:23 ` Kaelyn via Guix-patches via
2022-06-17  1:51 ` [bug#55231] [PATCH v2 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
2022-06-17 20:34   ` [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Ludovic Courtès
2022-06-18 19:11 ` [bug#55231] [PATCH v3 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
2022-06-18 19:11   ` [bug#55231] [PATCH v3 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’ Brian Cully via Guix-patches via
2022-06-18 20:34     ` Maxime Devos
2022-06-18 20:43       ` Brian Cully via Guix-patches via
2022-06-18 22:34         ` Maxime Devos
2022-06-18 23:11           ` Brian Cully via Guix-patches via
2022-06-19 12:05             ` Maxime Devos
2022-06-21 12:34 ` [bug#55231] [PATCH v1] initrd: Allow extra search paths with ‘initrd-extra-module-paths’ Kaelyn via Guix-patches via
2022-06-24 13:28 ` [bug#55231] [PATCH v4 1/2] Allows copying of out-of-tree modules to the Linux initrd Brian Cully via Guix-patches via
2022-06-24 13:28   ` [bug#55231] [PATCH v4 2/2] doc: ‘initrd-modules’ will search ‘kernel-loadable-modules’ Brian Cully via Guix-patches via

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).