bug#30760: guix system init broken on non GuixSD

bug#30760: guix system init broken on non GuixSD

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

`guix system init` seems to be broken for non GuixSD distirbutions:
When I tried it on openSUSE:

# guix system --no-bootloader init /Devel/git/guix-config/config.scm /mnt/mnt/
;;; note: source file /Devel/extra/gnu/packages/connman.scm
;;;       newer than compiled /root/.config/guix/latest/gnu/packages/connman.go
;;; note: source file /Devel/extra/gnu/packages/connman.scm
;;;       newer than compiled /usr/lib64/guile/2.2/site-ccache/gnu/packages/connman.go
;;; note: source file /Devel/extra/gnu/packages/connman.scm
;;;       newer than compiled /usr/lib64/guile/2.2/site-ccache/gnu/packages/connman.go
guix system: error: open-file: No such file or directory: "/run/booted-system/kernel/lib/modules/4.15.6-1-default/modules.alias"

4.15.6-1-default is version of my running kernel, but not defined as package - it is not expected to be used for guix call.

/run/booted-system/ is specific for GuixSD.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#30760: guix system init broken on non GuixSD

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

Hi Ludo,

there's a problem with check-device-initrd-modules: on "guix system init"
it doesn't get linux-module-directory and doesn't pass it on to matching-modules.
matching-modules then eventually defaults to (current-alias-file) - which is not
found on a non-GuixSD system.

Would it be possible to get rid of the defaults in gnu/build/linux-modules.scm ?
I don't think those are safe or useful for our requirements.  I've had to work
around those before.

check-initrd-modules could use the initrd's new kernel modules to find
out which modules to include (after all).

Then we could also check the dependencies directly in the new Linux kernel
modules and all in all it would be safer.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#30760: guix system init broken on non GuixSD

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> there's a problem with check-device-initrd-modules: on "guix system init"
> it doesn't get linux-module-directory and doesn't pass it on to matching-modules.
> matching-modules then eventually defaults to (current-alias-file) - which is not
> found on a non-GuixSD system.

Yeah.

> Would it be possible to get rid of the defaults in gnu/build/linux-modules.scm ?
> I don't think those are safe or useful for our requirements.  I've had to work
> around those before.
>
> check-initrd-modules could use the initrd's new kernel modules to find
> out which modules to include (after all).
>
> Then we could also check the dependencies directly in the new Linux kernel
> modules and all in all it would be safer.

It would be safer indeed, but we’d have to build the kernel and
everything before we can make a diagnostic.  That would lead to a weird
user experience, similar to what we currently see with grafts (things
are built/downloader, and later on you get a message about what’s going
to be built.)

The current tradeoff is to make that diagnostic based on the running
kernel, even if it’s an approximation.

If that’s fine with you I’d like to fix this bug with the conservative
patch below.

Thoughts?

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2743 bytes --]

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4fe673cca..8cae4fb63 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -40,6 +40,7 @@
             current-module-debugging-port
 
             device-module-aliases
+            current-alias-file
             known-module-aliases
             matching-modules))
 
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 1eb5f5130..16a8c4375 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -353,17 +353,27 @@ loaded at boot time in the order in which they appear."
 (define (check-device-initrd-modules device linux-modules location)
   "Raise an error if DEVICE needs modules beyond LINUX-MODULES to operate.
 DEVICE must be a \"/dev\" file name."
-  (let ((modules (delete-duplicates
-                  (append-map matching-modules
-                              (device-module-aliases device)))))
-    (unless (every (cute member <> linux-modules) modules)
-      (raise (condition
-              (&message
-               (message (format #f (G_ "you may need these modules \
+  (define aliases
+    ;; Attempt to load 'modules.alias' from the current kernel, assuming we're
+    ;; on GuixSD, and assuming that corresponds to the kernel we'll be
+    ;; installing.  Skip the whole thing if that file cannot be read.
+    (catch 'system-error
+      (lambda ()
+        (known-module-aliases))
+      (const #f)))
+
+  (when aliases
+    (let ((modules (delete-duplicates
+                    (append-map (cut matching-modules <> aliases)
+                                (device-module-aliases device)))))
+      (unless (every (cute member <> linux-modules) modules)
+        (raise (condition
+                (&message
+                 (message (format #f (G_ "you may need these modules \
 in the initrd for ~a:~{ ~a~}")
-                                device modules)))
-              (&fix-hint
-               (hint (format #f (G_ "Try adding them to the
+                                  device modules)))
+                (&fix-hint
+                 (hint (format #f (G_ "Try adding them to the
 @code{initrd-modules} field of your @code{operating-system} declaration, along
 these lines:
 
@@ -373,8 +383,8 @@ these lines:
    (initrd-modules (append (list~{ ~s~})
                            %base-initrd-modules)))
 @end example\n")
-                             modules)))
-              (&error-location
-               (location (source-properties->location location))))))))
+                               modules)))
+                (&error-location
+                 (location (source-properties->location location)))))))))
 
 ;;; linux-initrd.scm ends here

bug#30760: guix system init broken on non GuixSD

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

[huge build]
> The current tradeoff is to make that diagnostic based on the running
> kernel, even if it’s an approximation.

Ah, good point.

> If that’s fine with you I’d like to fix this bug with the conservative
> patch below.

Sure, looks good.

While we are approximating we could also in a later version fall back to
(the host system's) "`cat /proc/sys/kernel/modprobe` --showconfig" - it
could be used to find aliases.

But maybe that would make it brittle.  Hmm...



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#30760: guix system init broken on non GuixSD

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> [huge build]
>> The current tradeoff is to make that diagnostic based on the running
>> kernel, even if it’s an approximation.
>
> Ah, good point.
>
>> If that’s fine with you I’d like to fix this bug with the conservative
>> patch below.
>
> Sure, looks good.

Pushed as 8d5c14edf5a6d01f859b1aa00c836ffdb5ddecf4.

> While we are approximating we could also in a later version fall back to
> (the host system's) "`cat /proc/sys/kernel/modprobe` --showconfig" - it
> could be used to find aliases.
>
> But maybe that would make it brittle.  Hmm...

Yeah, I don’t think it’d make a big difference.

BTW, we should add a ‘--skip-checks’ option to ‘guix system’ so that
users can skip those checks.  That’d provide an escape hatch in case
‘check-device-initrd-modules’ makes the wrong diagnostic.

Thoughts?

Ludo’.

bug#30760: guix system init broken on non GuixSD

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

On Sat, Mar 10, 2018 at 12:19:52AM +0100, Ludovic Courtès wrote:
>Danny Milosavljevic <dannym@scratchpost.org> skribis:
>
>> [huge build]
>>> The current tradeoff is to make that diagnostic based on the running
>>> kernel, even if it’s an approximation.
>>
>> Ah, good point.
>>
>>> If that’s fine with you I’d like to fix this bug with the conservative
>>> patch below.
>>
>> Sure, looks good.
>
>Pushed as 8d5c14edf5a6d01f859b1aa00c836ffdb5ddecf4.

I'm afraid that now it leads to:

Backtrace:
         12 (primitive-load "/usr/bin/guix")
In guix/ui.scm:
 1501:12 11 (run-guix-command _ . _)
In ice-9/boot-9.scm:
   829:9 10 (catch _ _ #<procedure 7fa007c66420 at guix/ui.scm:552…> …)
   829:9  9 (catch _ _ #<procedure 7fa007c66438 at guix/ui.scm:660…> …)
In guix/scripts/system.scm:
  1180:8  8 (_)
  1052:6  7 (process-action _ _ _)
In guix/store.scm:
 1443:24  6 (run-with-store _ _ #:guile-for-build _ #:system _ # _)
In guix/scripts/system.scm:
 1065:13  5 (_ _)
   764:4  4 (perform-action init #<<operating-system> kernel: #<pa…> …)
In srfi/srfi-1.scm:
   640:9  3 (for-each #<procedure 55b8f874b9c0 at guix/scripts/sys…> …)
In gnu/system/linux-initrd.scm:
   360:4  2 (check-device-initrd-modules "/dev/nvme0n1p2" ("ahci" …) …)
In ice-9/boot-9.scm:
   829:9  1 (catch system-error #<procedure 7fa004522f60 at gnu/sy…> …)
In gnu/system/linux-initrd.scm:
   361:6  0 (_)

gnu/system/linux-initrd.scm:361:6: known-module-aliases: unbound variable


This is part of my config:

(initrd (lambda (file-system . rest)
          (raw-initrd file-systems
                      #:linux linux-x1-sw1
                      #:linux-modules '()
                      #:helper-packages '(linux-firmware-initrd-x1-sw1)
                      #:mapped-devices mapped-devices)))


I don't have any modules to be loaded in initrd, kernel is compiled
using my configuration which fits my needs and follows the HW it will run on.

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#30760: guix system init broken on non GuixSD

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

Hi Ludo,

> BTW, we should add a ‘--skip-checks’ option to ‘guix system’ so that
> users can skip those checks.  That’d provide an escape hatch in case
> ‘check-device-initrd-modules’ makes the wrong diagnostic.

Yeah, good idea!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#30760: guix system init broken on non GuixSD

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

On Sun, Mar 11, 2018 at 10:38:18PM +0100, Ludovic Courtès wrote:
>Tomáš Čech <sleep_walker@gnu.org> skribis:
>
>> In ice-9/boot-9.scm:
>>    829:9  1 (catch system-error #<procedure 7fa004522f60 at gnu/sy…> …)
>> In gnu/system/linux-initrd.scm:
>>    361:6  0 (_)
>>
>> gnu/system/linux-initrd.scm:361:6: known-module-aliases: unbound variable
>
>My bad!  Danny eventually fixed it in
>0803ddf2677ead5e9d8ef698316125e0c8b9c998.

I'm afraid this is still not correct.

# guix system init config.scm /mnt/mnt/
...
config.scm:64:9: error: you may need these modules in the initrd for /dev/nvme0n1p2: shpchp
hint: Try adding them to the `initrd-modules' field of your `operating-system' declaration, along these lines:

      (operating-system
        ;; ...
        (initrd-modules (append (list "shpchp")
                                %base-initrd-modules)))

I don't have `shpchp` as a module as I have it compiled into kernel
directly. Can I somehow disable the check?

Thanks.

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#30760: guix system init broken on non GuixSD

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

> I'm afraid this is still not correct.
> 
> # guix system init config.scm /mnt/mnt/
> ...
> config.scm:64:9: error: you may need these modules in the initrd for /dev/nvme0n1p2: shpchp
> hint: Try adding them to the `initrd-modules' field of your `operating-system' declaration, along these lines:
> 
>       (operating-system
>         ;; ...
>         (initrd-modules (append (list "shpchp")
>                                 %base-initrd-modules)))
> 
> I don't have `shpchp` as a module as I have it compiled into kernel
> directly. Can I somehow disable the check?

I think it's a good idea to add a command-line switch that disables the check.

But then people will just disable the check always and it won't improve until
it's correct.  It's still a good idea to give people the choice.

@Ludo: It would also be great to have a command-line switch to check the slow,
correct, way.  We'd also have to check modules.builtin of the new system's initrd
- but we'd do it only when the option is passed :)

I suggest to change it to:

> # guix system init config.scm /mnt/mnt/
> ...
> config.scm:64:9: WARNING: you may need these modules in the initrd for /dev/nvme0n1p2: shpchp
                   ^^^ not error
> hint: Try adding them to the `initrd-modules' field of your `operating-system' declaration, along these lines:
> 
>       (operating-system
>         ;; ...
>         (initrd-modules (append (list "shpchp")
>                                 %base-initrd-modules)))
>
> If you think this warning is mistaken, invoke guix again with the option
> --enable-paranoid-initrd-checks to be on the safe side, or with the option
> --skip-initrd-checks to continue regardless.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#30760: guix system init broken on non GuixSD

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

Or maybe to this:

> > # guix system init config.scm /mnt/mnt/
> > ...
> > config.scm:64:9: WARNING: you may need these modules in the initrd for /dev/nvme0n1p2: shpchp  
>                    ^^^ not error
> > hint: Try adding them to the `initrd-modules' field of your `operating-system' declaration, along these lines:
> > 
> >       (operating-system
> >         ;; ...
> >         (initrd-modules (append (list "shpchp")
> >                                 %base-initrd-modules)))
> >
> > (sleeps 5 s)
> > (builds entire system)
> > (checks the slow, correct way at the end - when everything was built already anyway)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#30760: guix system init broken on non GuixSD

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1941 bytes --]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> I'm afraid this is still not correct.
>> 
>> # guix system init config.scm /mnt/mnt/
>> ...
>> config.scm:64:9: error: you may need these modules in the initrd for /dev/nvme0n1p2: shpchp
>> hint: Try adding them to the `initrd-modules' field of your `operating-system' declaration, along these lines:
>> 
>>       (operating-system
>>         ;; ...
>>         (initrd-modules (append (list "shpchp")
>>                                 %base-initrd-modules)))
>> 
>> I don't have `shpchp` as a module as I have it compiled into kernel
>> directly. Can I somehow disable the check?

Exactly what I feared.  ;-)

> I think it's a good idea to add a command-line switch that disables the check.
>
> But then people will just disable the check always and it won't improve until
> it's correct.  It's still a good idea to give people the choice.
>
> @Ludo: It would also be great to have a command-line switch to check the slow,
> correct, way.  We'd also have to check modules.builtin of the new system's initrd
> - but we'd do it only when the option is passed :)
>
> I suggest to change it to:
>
>> # guix system init config.scm /mnt/mnt/
>> ...
>> config.scm:64:9: WARNING: you may need these modules in the initrd for /dev/nvme0n1p2: shpchp
>                    ^^^ not error

I thought about making it a warning rather than an error back then, but
thought that it wouldn’t work well: the warning would immediately go
off-screen as build logs start scrolling by.

Thus I took the optimistic view that false positives like the one Tomáš
experienced should be rare because usually init/reconfigure are used on
GuixSD, with a kernel config very close to the target config.

Nevertheless, the risk of false-positives obviously exists, hence the
need for an escape hatch.

What about the attached patch?

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 4059 bytes --]

diff --git a/doc/guix.texi b/doc/guix.texi
index d3a7908f9..bcea89e07 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20458,6 +20458,16 @@ of the image size as a function of the size of the system declared in
 Make @var{file} a symlink to the result, and register it as a garbage
 collector root.
 
+@item --skip-checks
+Skip pre-installation safety checks.
+
+By default, @command{guix system init} and @command{guix system
+reconfigure} perform safety checks: they make sure the file systems that
+appear in the @code{operating-system} declaration actually exist
+(@pxref{File Systems}), and that any Linux kernel modules that may be
+needed at boot time are listed in @code{initrd-modules} (@pxref{Initial
+RAM Disk}).  Passing this option skips these tests altogether.
+
 @item --on-error=@var{strategy}
 Apply @var{strategy} when an error occurs when reading @var{file}.
 @var{strategy} may be one of the following:
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index acfccce96..f0c4a2ba1 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -733,7 +733,8 @@ and TARGET arguments."
                       (#$installer #$bootloader #$device #$target))))))
 
 (define* (perform-action action os
-                         #:key install-bootloader?
+                         #:key skip-safety-checks?
+                         install-bootloader?
                          dry-run? derivations-only?
                          use-substitutes? bootloader-target target
                          image-size file-system-type full-boot?
@@ -750,7 +751,10 @@ When DERIVATIONS-ONLY? is true, print the derivation file name(s) without
 building anything.
 
 When GC-ROOT is a path, also make that path an indirect root of the build
-output when building a system derivation, such as a disk image."
+output when building a system derivation, such as a disk image.
+
+When SKIP-SAFETY-CHECKS? is true, skip the file system and initrd module
+static checks."
   (define println
     (cut format #t "~a~%" <>))
 
@@ -760,7 +764,8 @@ output when building a system derivation, such as a disk image."
   ;; Check whether the declared file systems exist.  This is better than
   ;; instantiating a broken configuration.  Assume that we can only check if
   ;; running as root.
-  (when (memq action '(init reconfigure))
+  (when (and (not skip-safety-checks?)
+             (memq action '(init reconfigure)))
     (check-mapped-devices os)
     (when (zero? (getuid))
       (check-file-system-availability (operating-system-file-systems os))
@@ -933,6 +938,8 @@ Some ACTIONS support additional ARGS.\n"))
       --expose=SPEC      for 'vm', expose host file system according to SPEC"))
   (display (G_ "
       --full-boot        for 'vm', make a full boot sequence"))
+  (display (G_ "
+      --skip-checks      skip file system and initrd module safety checks"))
   (newline)
   (display (G_ "
   -h, --help             display this help and exit"))
@@ -974,6 +981,9 @@ Some ACTIONS support additional ARGS.\n"))
          (option '("full-boot") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'full-boot? #t result)))
+         (option '("skip-checks") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'skip-safety-checks? #t result)))
 
          (option '("share") #t #f
                  (lambda (opt name arg result)
@@ -1067,6 +1077,8 @@ resulting from command-line parsing."
                              #:derivations-only? (assoc-ref opts
                                                             'derivations-only?)
                              #:use-substitutes? (assoc-ref opts 'substitutes?)
+                             #:skip-safety-checks?
+                             (assoc-ref opts 'skip-safety-checks?)
                              #:file-system-type (assoc-ref opts 'file-system-type)
                              #:image-size (assoc-ref opts 'image-size)
                              #:full-boot? (assoc-ref opts 'full-boot?)

bug#30760: guix system init broken on non GuixSD

[Ludovic Courtès]

Hi!

ludo@gnu.org (Ludovic Courtès) skribis:

> Nevertheless, the risk of false-positives obviously exists, hence the
> need for an escape hatch.
>
> What about the attached patch?

Pushed as 61b1dbbdcd87f6b37d6b87715a9a4da4e63485ab.

Thanks,
Ludo’.

bug#30760: guix system init broken on non GuixSD

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

On Mon, Mar 12, 2018 at 01:24:37PM +0100, Danny Milosavljevic wrote:
>> I'm afraid this is still not correct.
>>
>> # guix system init config.scm /mnt/mnt/
>> ...
>> config.scm:64:9: error: you may need these modules in the initrd for /dev/nvme0n1p2: shpchp
>> hint: Try adding them to the `initrd-modules' field of your `operating-system' declaration, along these lines:
>>
>>       (operating-system
>>         ;; ...
>>         (initrd-modules (append (list "shpchp")
>>                                 %base-initrd-modules)))
>>
>> I don't have `shpchp` as a module as I have it compiled into kernel
>> directly. Can I somehow disable the check?
>
>I think it's a good idea to add a command-line switch that disables the check.
>
>But then people will just disable the check always and it won't improve until
>it's correct.  It's still a good idea to give people the choice.

Just small note - In my case I always run `system build` before
`system init` so I don't mind having any deeper analysis based on code
and configuration as long as it is correct. Maybe more people is using
same approach.

Best regards,

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]