Hi Maxim,

I'd "debug" the issue trying to compare my Debian system config with
yours since I'm also using a BTRFS RAID1 filesystem on LUKS.

I've still not unplugged one of the two disks on mine to simulate a
drive failure, Soon™ I'd like to test this condition... but it's a
busy machine so I don't know when.

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

>>> Ideally, GRUB (or is it our boot script?)
>>
>> Since the end result is your system entered "grub rescue" mode AFAIU
>> it's a GRUB issue
>
> Yeah, it looks like it.  The grub.cfg file only has basic things in it,
> nothing that could explain the failure.

Please could you also provide the result of "lsblk -f"?

This is (part of) my disks layout:

--8<---------------cut here---------------start------------->8---

sdc                                                                                  
├─sdc1                                                                               
├─sdc2 vfat                      F6D8-67E3                             470.8M     1% /boot/efi
├─sdc3 crypto_L                  e554b806-19ac-48b2-b521-b4e89839a756                
│ └─crypt_swap01
│      swap                      a43ce70c-dd35-47d8-a2ef-ef9d3c6d0885                [SWAP]
└─sdc4 crypto_L                  820bfdf7-46f7-46f5-8536-7e1b0f04e70e                
  └─crypt_btrfs01_03
       btrfs    btrfs_pool01     82afe97a-bb97-4b3d-90cb-93a058185b97                
sdd                                                                                  
├─sdd1                                                                               
├─sdd2                                                                               
├─sdd3 crypto_L                  960aa919-182b-4604-a8be-8477c86386cc                
│ └─crypt_swap02
│      swap                      3f8f6974-05a9-4047-993a-c4ccb27eaa1d                [SWAP]
└─sdd4 crypto_L                  c590c62e-6ac8-418c-9ea7-7ae9c79058c8                
  └─crypt_btrfs01_04
       btrfs    btrfs_pool01     82afe97a-bb97-4b3d-90cb-93a058185b97  802.3G    57% /mnt/btrfs

--8<---------------cut here---------------end--------------->8---

btrfs_pool01 is my BTRFS RAID1 filesystem, it includes /boot and /
(root) and is on two ancrypted LUKS partitions, as you can see.

Also, please what's your grub.cfg?

This is the config of a menuentry of mine:

--8<---------------cut here---------------start------------->8---

menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-82afe97a-bb97-4b3d-90cb-93a058185b97' {
	load_video
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_gpt
	insmod cryptodisk
	insmod luks
	insmod gcry_rijndael
	insmod gcry_rijndael
	insmod gcry_sha256
	insmod btrfs
	cryptomount -u c590c62e6ac8418c9ea77ae9c79058c8
	set root='cryptouuid/c590c62e6ac8418c9ea77ae9c79058c8'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='cryptouuid/c590c62e6ac8418c9ea77ae9c79058c8'  82afe97a-bb97-4b3d-90cb-93a058185b97
	else
	  search --no-floppy --fs-uuid --set=root 82afe97a-bb97-4b3d-90cb-93a058185b97
	fi
	echo	'Loading Linux 5.10.0-0.bpo.3-amd64 ...'
	linux	/debian_root/boot/vmlinuz-5.10.0-0.bpo.3-amd64 root=UUID=82afe97a-bb97-4b3d-90cb-93a058185b97 ro rootflags=subvol=debian_root ip=10.38.2.2::10.38.2.1:255.255.255.0:anemone:eth0:none quiet
	echo	'Loading initial ramdisk ...'
	initrd	/debian_root/boot/initrd.img-5.10.0-0.bpo.3-amd64
}

--8<---------------cut here---------------end--------------->8---

AFAIU this code (from the snippet above):

--8<---------------cut here---------------start------------->8---

	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='cryptouuid/c590c62e6ac8418c9ea77ae9c79058c8'  82afe97a-bb97-4b3d-90cb-93a058185b97
	else
	  search --no-floppy --fs-uuid --set=root 82afe97a-bb97-4b3d-90cb-93a058185b97
	fi

--8<---------------cut here---------------end--------------->8---

sets [1] the root GRUB env variable to the first found device containing
the UUID 82afe97a-bb97-4b3d-90cb-93a058185b97, that is the UUID of my
BTRFS filesystem

AFAIU (but still not tested) this means that if the device with UUID
c590c62[...] is missing the search ensures that GRUB will find the next
device containing the BTRFS filesystem identified by UUID 82afe97a[...]

WDYT?

[1] https://www.gnu.org/software/grub/manual/grub/grub.html#search

[...]

>> Can you please provide the output of the "ls" command and the "set"
>> command from the grub rescue shell?
>
> I'll post after rebooting.

OK thanks.

>> Also, please what is your /proc/cmdline (when Linux correcly boots)?
>
> --8<---------------cut here---------------start------------->8---
> BOOT_IMAGE=/@root/gnu/store/1c0dkkkv5vdnyp73gvcl9k1kym5jjm54-linux-libre-5.13.8/bzImage
> --root=/dev/mapper/cryptroot
> --system=/gnu/store/815481yf1kfacwgkh4aa11rlb3lm6gvi-system
> --load=/gnu/store/815481yf1kfacwgkh4aa11rlb3lm6gvi-system/boot quiet
> snd_hda_intel.dmic_detect=0 modprobe.blacklist=rtl8187
> --8<---------------cut here---------------end--------------->8---

This is mine (derived from the GRUB menu entry shown above):

--8<---------------cut here---------------start------------->8---

BOOT_IMAGE=/debian_root/boot/vmlinuz-5.10.0-0.bpo.3-amd64 root=UUID=82afe97a-bb97-4b3d-90cb-93a058185b97 ro rootflags=subvol=debian_root ip=10.38.2.2::10.38.2.1:255.255.255.0:anemone:eth0:none quiet

--8<---------------cut here---------------end--------------->8---

AFAIU using "root=UUID=..." is more robust than using the (possibly
missing) device mapper path.

[...]

Hope this helps.

Best regards, Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures