bug#70897: Guix system hangs on boot with LUKS root partition

bug#70897: Guix system hangs on boot with LUKS root partition

[Kaelyn via Bug reports for GNU Guix]

Hi Ludo' and others,

I recently updated my systems after finally finding https://issues.guix.gnu.org/70051 and seeing the issue I was having with booting with a non-root LUKS partition configured had been fixed. After updating to a commit past these two:

  49f82fca41 mapped-devices: luks: Specify modules needed at the top-level.
  6062339156 mapped-devices: <mapped-device-type> can specify modules to import.

I am now seeing a different error, which I am pretty sure is related to the module import changes in 49f82fca41. The error I get is about an unknown symbol "system*/tty" when the initramfs tries to prompt for a password to unlock the LUKS partition containing the root filesystem. I don't know how the module plumbing of Shepherd and the generated initramfs work, but I suspect the fix for Shepherd opening LUKS partition broke the import of system*/tty in the initramfs (for example, at the early REPL that booting my latest system generation ends up at, system*/tty is undefined initially, but after evaluating "(use-modules (gnu build file-systems))" system*/tty resolves to a procedure as exected--so the module is at least present in the initramfs). I have encountered this error with two different systems, and I believe the reproduction is simply trying to open a LUKS device without a keyfile so that a password prompt is necessary.

Cheers,
Kaelyn

bug#70897: Guix system hangs on boot with LUKS root partition

[Ludovic Courtès]

Hi Kaelyn,

Kaelyn <kaelyn.alexi@protonmail.com> skribis:

> I recently updated my systems after finally finding https://issues.guix.gnu.org/70051 and seeing the issue I was having with booting with a non-root LUKS partition configured had been fixed. After updating to a commit past these two:
>
>   49f82fca41 mapped-devices: luks: Specify modules needed at the top-level.
>   6062339156 mapped-devices: <mapped-device-type> can specify modules to import.
>
> I am now seeing a different error, which I am pretty sure is related
> to the module import changes in 49f82fca41. The error I get is about
> an unknown symbol "system*/tty" when the initramfs tries to prompt for
> a password to unlock the LUKS partition containing the root
> filesystem.

To be clear, you have both a LUKS-encrypted root and a non-root
LUKS-encrypted partition?

(FWIW I tested (1) with a LUKS-encrypted root, and (2) with a cleartext
root and LUKS-encrypted /home.  The bug you mention affected #2.)

Could you share your OS config or a relevant subset thereof?

> I don't know how the module plumbing of Shepherd and the generated
> initramfs work, but I suspect the fix for Shepherd opening LUKS
> partition broke the import of system*/tty in the initramfs (for
> example, at the early REPL that booting my latest system generation
> ends up at, system*/tty is undefined initially, but after evaluating
> "(use-modules (gnu build file-systems))" system*/tty resolves to a
> procedure as exected--so the module is at least present in the
> initramfs). I have encountered this error with two different systems,
> and I believe the reproduction is simply trying to open a LUKS device
> without a keyfile so that a password prompt is necessary.

Hmm.  Thanks for investigating!

Ludo’.

bug#70897: Guix system hangs on boot with LUKS root partition

[Kaelyn via Bug reports for GNU Guix]

Hi Ludo',

On Monday, May 13th, 2024 at 3:14 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> 
> 
> Hi Kaelyn,
> 
> Kaelyn kaelyn.alexi@protonmail.com skribis:
> 
> > I recently updated my systems after finally finding https://issues.guix.gnu.org/70051 and seeing the issue I was having with booting with a non-root LUKS partition configured had been fixed. After updating to a commit past these two:
> > 
> > 49f82fca41 mapped-devices: luks: Specify modules needed at the top-level.
> > 6062339156 mapped-devices: <mapped-device-type> can specify modules to import.
> > 
> > I am now seeing a different error, which I am pretty sure is related
> > to the module import changes in 49f82fca41. The error I get is about
> > an unknown symbol "system*/tty" when the initramfs tries to prompt for
> > a password to unlock the LUKS partition containing the root
> > filesystem.
> 
> 
> To be clear, you have both a LUKS-encrypted root and a non-root
> LUKS-encrypted partition?
> 
> (FWIW I tested (1) with a LUKS-encrypted root, and (2) with a cleartext
> root and LUKS-encrypted /home. The bug you mention affected #2.)

More accurately, I have one system that has a mirrored btrfs root with two LUKS-encrypted partitions (and a few quirks in the setup that make rebooting a bit tedious, such as grub slowly unlocking two drives, and a ZFS pool that has to be unlocked manually after boot), and one with a single LUKS-encrypted btrfs partition. I hit (2) on the first system about a month ago when I updated both, with the second system booting fine. I hit (1) on the second system when updating much more recently after seeing (2) was fixed, and hadn't tried rebooting the first system with the new generation.

> Could you share your OS config or a relevant subset thereof?

My full OS config is decidedly non-trivial, with parts (e.g. common services and user accounts) shared between host configurations. The mapped-devices and file-systems fragments for the two systems are below.

For the first system, with the mirrored btrfs root:

  (mapped-devices
    (list (mapped-device
            (source
              (uuid "7bcca55e-8a41-44a8-beab-2047eed0af41"))
            (target "cryptroot1")
            (type luks-device-mapping))
          (mapped-device
            (source
              (uuid "9472b8ae-c90c-4712-b90d-ca07602514d7"))
            (target "cryptroot2")
            (type luks-device-mapping))
          ))
  (file-systems
    (let ((rootfs (file-system
             (mount-point "/")
             (device "/dev/mapper/cryptroot1")
             (type "btrfs")
             (check? #f)
             (options "compress=zstd,subvol=@guix")
             (dependencies mapped-devices))))
    (cons* rootfs
           (file-system
             (mount-point "/boot/efi")
             (device (file-system-label "EFI"))
             (type "vfat")
             (mount-may-fail? #t)
             (dependencies mapped-devices))
           (file-system
             (mount-point "/gnu")
             (device "/dev/mapper/cryptroot1")
             (type "btrfs")
             (check? #f)
             (options "compress=zstd,subvol=@gnu_store")
             (dependencies (cons rootfs mapped-devices)))
           %base-file-systems))))


The second system, with the single-drive encrypted btrfs root:

  (mapped-devices
    (list (mapped-device
            (source
              (uuid "e6aaafc5-49cb-477b-a665-daf065611195"))
            (target "cryptroot1")
            (type luks-device-mapping))
          ))
  (file-systems
    (let ((rootfs (file-system
             (mount-point "/")
             (device "/dev/mapper/cryptroot1")
             (type "btrfs")
             (check? #f)
             (options "compress=zstd,subvol=@guix")
             (dependencies mapped-devices))))
    (cons* rootfs
           (file-system
             (mount-point "/boot/efi")
             (device (file-system-label "EFI"))
             (type "vfat")
             (mount-may-fail? #t)
             (dependencies mapped-devices))
           (file-system
             (mount-point "/gnu")
             (device "/dev/mapper/cryptroot1")
             (type "btrfs")
             (check? #f)
             (options "compress=zstd,subvol=@gnu_store")
             (dependencies (cons rootfs mapped-devices)))
           %common-file-systems))))


(Note the %common-file-systems is simply %base-file-systems plus a couple of NFS mounts from the first system, which are shared with several computers.)

For both computers, I make use of the 6.1 or 6.6 LTS kernels since I also use ZFS. When I hit (1), I eventually figured out where the hang during boot was happening by removing "quiet" from the kernel command line, which also caused shepherd to be more verbose (something I hadn't realized). When I hit (2), the boot process was still in the initrd due to failing to unlock and mount the root filesystem.

If there is any further information I can provide, please let me know.

Cheers,
Kaelyn

> 
> > I don't know how the module plumbing of Shepherd and the generated
> > initramfs work, but I suspect the fix for Shepherd opening LUKS
> > partition broke the import of system*/tty in the initramfs (for
> > example, at the early REPL that booting my latest system generation
> > ends up at, system*/tty is undefined initially, but after evaluating
> > "(use-modules (gnu build file-systems))" system*/tty resolves to a
> > procedure as exected--so the module is at least present in the
> > initramfs). I have encountered this error with two different systems,
> > and I believe the reproduction is simply trying to open a LUKS device
> > without a keyfile so that a password prompt is necessary.
> 
> 
> Hmm. Thanks for investigating!
> 
> Ludo’.

bug#70897: Guix system hangs on boot with LUKS root partition

[Kaelyn via Bug reports for GNU Guix]

Hi Ludo',

I wanted to let you know that I have figured out what the failure is, and it turned out to be a customization buried within some shared operating-system configuration that I had forgotten about (and wasn't finding because it was due to a missing import, and simply grepping for "system*/tty" didn't match the file that was missing the import). Thank you for your time and help, and sorry about the red herring of a bug report!

Cheers,
Kaelyn

bug#70897: Guix system hangs on boot with LUKS root partition

[Ludovic Courtès]

Hi,

Kaelyn <kaelyn.alexi@protonmail.com> skribis:

> I wanted to let you know that I have figured out what the failure is,
> and it turned out to be a customization buried within some shared
> operating-system configuration that I had forgotten about (and wasn't
> finding because it was due to a missing import, and simply grepping
> for "system*/tty" didn't match the file that was missing the
> import). Thank you for your time and help, and sorry about the red
> herring of a bug report!

Great, glad you found the problem!

Ludo’.