bug#19190: Cannot boot with encrypted root

bug#19190: Cannot boot with encrypted root

[Nikita Karetnikov]

[-- Attachment #1: Type: text/plain, Size: 2458 bytes --]

(Ludo suggested to report this as a bug, so this issue won’t get lost.)

While booting, GRUB prints this error:

error: file
‘/gnu/store/rz…-linux-libre-3.17.3/bzImage’ not found.

which is not surprising since the store is encrypted.

I’ve tried putting these lines after the “setparams” line in the GRUB
config (by hitting the ‘e’ key):

insmod luks
cryptomount hd0,gpt3
set root=crypto0

After pressing the ‘F10’ key, I get a password prompt.  And after that, a
bunch of warnings about ‘/dev/mapper/main’:

ext2fs_check_if_mount: Can’t check if filesystem is mounted due to
missing mtab file while determining whether /dev/mapper/main is mounted.
fsck.ext3: No such file or directory while trying to open
/dev/mapper/main
Possibly non-existent device?
‘fsck.ext3’ exited with code 8 on /dev/mapper/main; spawning REPL

In the REPL:

scheme@(guile-user)> ,use (gnu build linux-boot)
scheme@(guile-user)> (boot-system)
[…]
ERROR: In procedure mount:
ERROR: In procedure mount: Device or resource busy

My config is shown below:

(use-modules (gnu))

(define %linux-modules
  '(
    ;; cryptsetup/LUKS
    "dm-crypt.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko"))

(operating-system
  (host-name "test")
  (timezone "Europe/Paris")
  (locale "en_US.UTF-8")

  (bootloader (grub-configuration
               (device "/dev/sda")))

  (initrd (lambda (fs . args)
            (apply base-initrd fs
                   #:extra-modules %linux-modules
                   args)))

  (mapped-devices (list (mapped-device
                         (source "/dev/sda3")
                         (target "main")
                         (type luks-device-mapping))))

  (file-systems (cons* (file-system
                         (device "/dev/mapper/main")
                         (mount-point "/")
                         (type "ext3"))
                       (file-system
                         (device "boot")
                         (type 'label)
                         (mount-point "/boot")
                         (type "ext3"))
                       %base-file-systems))

  (users (list (user-account
                (name "test")
                (group "users")

                (supplementary-groups '("wheel"
                                        "audio" "video"))
                (home-directory "/home/test")))))

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#19190: Cannot boot with encrypted root

[Ludovic Courtès]

Thanks for the report.  Let’s address the initrd/boot issue first.

Nikita Karetnikov <nikita@karetnikov.org> skribis:

> After pressing the ‘F10’ key, I get a password prompt.

Any message after entering the password prompt?

> And after that, a bunch of warnings about ‘/dev/mapper/main’:
>
> ext2fs_check_if_mount: Can’t check if filesystem is mounted due to
> missing mtab file while determining whether /dev/mapper/main is mounted.
> fsck.ext3: No such file or directory while trying to open
> /dev/mapper/main
> Possibly non-existent device?
> ‘fsck.ext3’ exited with code 8 on /dev/mapper/main; spawning REPL

It looks like ‘cryptsetup open’ simply failed, and ‘boot-system’
currently ignores that error, so it goes up to the file system check as
shown above, which in turn fails and gets us to the debugger.

Perhaps you could try running ‘cryptsetup’ by hand from the REPL.
That’s going to be a bit tedious; something like:

  (use-modules (srfi srfi-1) (ice-9 ftw))
  (define cs (find (lambda (file)
                     (string-contains file "cryptsetup"))
                   (scandir "/gnu/store")))
  (system* (string-append "/gnu/store/" cs "/bin/cryptsetup")
           "open" "--type" "luks" "/dev/whatever" "main")

Then, please report the return value of ‘system*’, and check

  (file-exists? "/dev/mapper/main")

Please also report any messages.

Thanks in advance!

Ludo’.

bug#19190: Cannot boot with encrypted root

[Duncan Keall]

Here's what I've found using a very similar setup to Nikita and after
receiving the same error messages.

> Any message after entering the password prompt?

GRUB prints a single message "Slot 0 opened" and then it's into the
kernel boot logs. The first related message appears to be the
"ext2fs_check_if_mount..." one.

> Perhaps you could try running ‘cryptsetup’ by hand from the REPL.

I needed to replace "/bin/cryptsetup" with "/sbin/cryptsetup". After
that, cryptsetup accepted a password and printed these errors:

  device-mapper: table 253:0: crypt: Error allocating crypto tfm
  device-mapper: ioctl: error adding target to table
  device-mapper: reload ioctl on  failed: No such file or directory
  Failed to open temporary keystore device.
  device-mapper: remove ioctl on temporary-cryptsetup-149 failed: No
  such device or address

The return value was 256, and afterwards /dev/mapper/main still did not
exist.

bug#19190: Cannot boot with encrypted root

[Ludovic Courtès]

I noticed an obvious issue in (gnu build linux-boot), which is that
device-mapping (the #:pre-mount argument) would be performed after
‘mount-root-file-system’ has been called.

Commit 67979e4 fixes that.

To fix it, we’ll have to update ‘guix-devel’ in (gnu packages
package-management), and to regenerate a VM image that uses it.  I can
help with that.

Ludo’.

bug#19190: Cannot boot with encrypted root

[Ludovic Courtès]

Hi,  :-)

Duncan Keall <duncan@duncankeall.com> skribis:

>> Perhaps you could try running ‘cryptsetup’ by hand from the REPL.
>
> I needed to replace "/bin/cryptsetup" with "/sbin/cryptsetup". After
> that, cryptsetup accepted a password and printed these errors:
>
>   device-mapper: table 253:0: crypt: Error allocating crypto tfm
>   device-mapper: ioctl: error adding target to table
>   device-mapper: reload ioctl on  failed: No such file or directory
>   Failed to open temporary keystore device.
>   device-mapper: remove ioctl on temporary-cryptsetup-149 failed: No
>   such device or address

I suspect this is due to missing modules.

Does your initrd configuration have the modules Nikita’s configuration
shows?

Thanks for testing so quickly!

Ludo’.

bug#19190: Cannot boot with encrypted root

[Duncan Keall]

> Does your initrd configuration have the modules Nikita’s configuration
> shows?

Here are the configurations I have tested which all result in the errors
posted above:

  ("dm-crypt.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
  ("dm-crypt.ko" "cryptd.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
  ("dm-crypt.ko" "cryptd.ko" "crc32-pclmul.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")

I could also add "aesni-intel.ko" but it resulted in a different set of
kernel errors on boot:

  aesni_intel: Unknown symbol ablk_decrypt (err 0)
  aesni_intel: Unknown symbol lrw_free_table (err 0)
  aesni_intel: Unknown symbol ablk_set_key (err 0)
  ...
  ERROR: In procedure load-linux-module: Unknown error -1

For reference, here are all the modules loaded on my current system
which might be related:

  aes_x86_64 aesni_intel af_alg algif_skcipher arc4 ccm crc32_pclmul
  crc32c_intel crc_t10dif crct10dif_common crct10dif_pclmul cryptd ctr
  dm_crypt dm_mod gf128mul ghash_clmulni_intel lrw xor

bug#19190: Cannot boot with encrypted root

[Ludovic Courtès]

Duncan Keall <duncan@duncankeall.com> skribis:

> Here are the configurations I have tested which all result in the errors
> posted above:
>
>   ("dm-crypt.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>   ("dm-crypt.ko" "cryptd.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>   ("dm-crypt.ko" "cryptd.ko" "crc32-pclmul.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>
> I could also add "aesni-intel.ko" but it resulted in a different set of
> kernel errors on boot:
>
>   aesni_intel: Unknown symbol ablk_decrypt (err 0)
>   aesni_intel: Unknown symbol lrw_free_table (err 0)
>   aesni_intel: Unknown symbol ablk_set_key (err 0)
>   ...
>   ERROR: In procedure load-linux-module: Unknown error -1

Presumably that’s because one of its prerequisites hasn’t been loaded
yet.

The difficulty is that modules need to be listed in topological order:
things that aesni_intel.ko depends on must appear before aesni_intel.ko.

On my machine, I see:

--8<---------------cut here---------------start------------->8---
$ lsmod|grep aes
aesni_intel           165373  3 
ablk_helper            13597  1 aesni_intel
cryptd                 20359  4 ghash_clmulni_intel,aesni_intel,ablk_helper
lrw                    13286  1 aesni_intel
glue_helper            13990  1 aesni_intel
aes_x86_64             17131  1 aesni_intel
--8<---------------cut here---------------end--------------->8---

So the right order for these would be:

  cryptd
  aes_x86_64
  ablk_helper
  lrw
  aesni_intel

We’ll have to change the initrd build to figure that out automatically,
but in the meantime, we have to sort them by hand.

Ludo’.

bug#19190: Cannot boot with encrypted root

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Duncan Keall <duncan@duncankeall.com> skribis:
>
>> Here are the configurations I have tested which all result in the errors
>> posted above:
>>
>>   ("dm-crypt.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>>   ("dm-crypt.ko" "cryptd.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>>   ("dm-crypt.ko" "cryptd.ko" "crc32-pclmul.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>>
>> I could also add "aesni-intel.ko" but it resulted in a different set of
>> kernel errors on boot:
>>
>>   aesni_intel: Unknown symbol ablk_decrypt (err 0)
>>   aesni_intel: Unknown symbol lrw_free_table (err 0)
>>   aesni_intel: Unknown symbol ablk_set_key (err 0)
>>   ...
>>   ERROR: In procedure load-linux-module: Unknown error -1
>
> Presumably that’s because one of its prerequisites hasn’t been loaded
> yet.
>
> The difficulty is that modules need to be listed in topological order:

This particular problem was fixed in 0e704a2 a couple of weeks ago.

For testing, you would need to start from an image generated from
current master with:

  ./pre-inst-env \
    guix system disk-image --image-size=800MiB gnu/system/install.scm

Let me know if anything else is needed.

Ludo’.

bug#19190: Cannot boot with encrypted root

[Duncan Keall]

Thanks for the update, Ludovic!

I've just tested another install with a fresh image from master, using
as similar setup as possible to before.

The boot process still fails at mounting the root filesystem (as
expected):

  fsck.ext4: No such file or directory while trying to open /dev/mapper/main
  Possibly non-existent device?

However I found that cryptsetup was missing from the store during the
early-boot REPL, so I wasn't able to test manually mounting the
encrypted filesystem. 

Nothing in the commit history since v0.8 jumped out as being
responsible, so I'm assuming I've missed something obvious during
install. I'll have to keep looking!

bug#19190: Cannot boot with encrypted root

[Ludovic Courtès]

Duncan Keall <duncan@duncankeall.com> skribis:

> The boot process still fails at mounting the root filesystem (as
> expected):
>
>   fsck.ext4: No such file or directory while trying to open /dev/mapper/main
>   Possibly non-existent device?

It took us a while but this part is now fixed with cc0e575, and commit
07779d0 adds dm-crypt.ko and xts.ko and their dependencies to the
default initrd.

I’m closing this bug, but if you have time and interest, you’re welcome
to confirm whether this works for you.

Thanks!

Ludo’.