bug#40999: GRUB prevents booting a degraded RAID1 array atop LUKS

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

Hello Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:

> Hello Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>>> On a system where:
>>>
>>> 1) Each disks comprising the array is fully LUKS encrypted
>>> 2) Each mapped disk is made part of a Btrfs RAID1 array
>>>
>>> When attempting to boot the system after pulling out (in BIOS or using
>>> the cable) the drive to simulate a complete disk failure, GRUB hangs,
>>> prompting for the LUKS password of the disappeared drive and
>>> (unsurprisingly) failing to open it.
>
> [...]
>
>> Ideally, GRUB (or is it our boot script?)
>
> Since the end result is your system entered "grub rescue" mode AFAIU
> it's a GRUB issue

Yeah, it looks like it.  The grub.cfg file only has basic things in it,
nothing that could explain the failure.

>> should be smart enough to realize that oh, that's Btrfs RAID1, it
>> ought to work in degraded mode, so let's keep going.
>
> I (still) don't have a Guix System to test your setup and (try to) patch
> thing up, so we need more info to debug the situation.

I believe the basic recipe to reproduce is there:

1. Partition two drives like so (GPT with 2MiB BIOS boot):

$ sudo sfdisk -l /dev/sda
Disk /dev/sda: 931.53 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WDC WD1002FAEX-0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: B5BB7BA4-23A3-4E7C-87BB-8339B02C5905

Device     Start        End    Sectors   Size Type
/dev/sda1   2048       6143       4096     2M BIOS boot
/dev/sda2   6144 1953523711 1953517568 931.5G Linux filesystem

$ sudo sfdisk -l /dev/sdb
Disk /dev/sdb: 931.53 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WDC WD1002FAEX-0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 45C58C18-7B39-A745-B22F-6A2321FB1999

Device     Start        End    Sectors   Size Type
/dev/sdb1   2048       6143       4096     2M BIOS boot
/dev/sdb2   6144 1953523711 1953517568 931.5G Linux filesystem

2. LUKS encrypt the whole 2nd (main) partition of each drive.

3. Format the mapped devices as Btrfs RAID1.

4. Reconfigure a Guix system on top of that (see a config snippet below)

5. Disconnect one of the two drives and reboot.

6. Contemplate the failure to get past GRUB.

> Can you please provide the output of the "ls" command and the "set"
> command from the grub rescue shell?

I'll post after rebooting.

> Also, please what is your /proc/cmdline (when Linux correcly boots)?

--8<---------------cut here---------------start------------->8---
BOOT_IMAGE=/@root/gnu/store/1c0dkkkv5vdnyp73gvcl9k1kym5jjm54-linux-libre-5.13.8/bzImage
--root=/dev/mapper/cryptroot
--system=/gnu/store/815481yf1kfacwgkh4aa11rlb3lm6gvi-system
--load=/gnu/store/815481yf1kfacwgkh4aa11rlb3lm6gvi-system/boot quiet
snd_hda_intel.dmic_detect=0 modprobe.blacklist=rtl8187
--8<---------------cut here---------------end--------------->8---

The system config relevant sections are:

--8<---------------cut here---------------start------------->8---
(operating-system
    (host-name "hurd")
    (timezone "America/Montreal")
    (keyboard-layout (keyboard-layout "dvorak"))
    (bootloader (bootloader-configuration
                 (bootloader grub-bootloader)
                 (target "/dev/sda")
                 (terminal-outputs '(console))
		 (keyboard-layout keyboard-layout)))
    (kernel-arguments '("quiet" "snd_hda_intel.dmic_detect=0"
                        "modprobe.blacklist=rtl8187"))
    (mapped-devices
     (list (mapped-device
            (source "/dev/sda2")
            (target "cryptroot")
            (type luks-device-mapping))
           (mapped-device
            (source "/dev/sdb2")
            (target "cryptroot-mirror")
            (type luks-device-mapping))
           (mapped-device
            (source "/dev/sdc2")
            (target "cryptroot-mirror2")
            (type luks-device-mapping))))

    ;; Note: Using any of the LUKS encrypted drives exposed under
    ;; /dev/mapper is enough to reference the Btrfs RAID-1 array,
    ;; since the 'btrfs device scan' command is executed in the init
    ;; RAM disk and takes care of assembling the array.
    (file-systems (cons* (file-system
                           (mount-point "/")
                           (device "/dev/mapper/cryptroot")
                           (type "btrfs")
			   (options (alist->file-system-options
                                     (cons '("subvol" . "@root")
                                           %common-btrfs-options)))
			   (dependencies mapped-devices))
                         (file-system
                           (device "/dev/mapper/cryptroot")
                           (mount-point "/home")
                           (type "btrfs")
                           (options (alist->file-system-options
                                     (cons '("subvol" . "@home")
                                           %common-btrfs-options)))
                           (dependencies mapped-devices))
                         (file-system
                           (device "/dev/mapper/cryptroot")
                           (mount-point "/data")
                           (type "btrfs")
                           (options (alist->file-system-options
                                     (cons '("subvol" . "@data")
                                           %common-btrfs-options)))
                           (dependencies mapped-devices))
                         %base-file-systems))
   [...]
--8<---------------cut here---------------end--------------->8---

Thanks,

Maxim