cannot boot with BTRFS in degraded mode

cannot boot with BTRFS in degraded mode

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2757 bytes --]

Hi Guix!

Yesterday I had to physically replace a failed disk on milano-guix-1
(one of Guix build machines), that disk was part of a BTRFS RAID10
multi disk array and now the machine is unbootable

The BTRFS RAID10 array was made of 6 disks and was running well, some
days ago Christopher Baines found that the 5th disk (/dev/sde) of that
array failed and was able to remount it in degraded mode in order to
re-balance the array and go on working without data loss

Unfortunately I was not able to perform a "btrfs replace..."  since
adding a new disk (we have spare slots) was not detected by the
kernel... HP ProLiant Smart Array is not so smart after all (aka bye bye
hot swapping of disks) :-S...

So I had to reboot the server and enter the config tool, added the new
drive as a new Smart Array logical volume (RAID0 with 1 drive) [1] and
removed the failed logical volume

The problem now is that the boot process stops when trying to mount the
BTRFS filesystem, the error is:

--8<---------------cut here---------------start------------->8---
BTRFS error (device sda3): devid 5 uuid [omissis] is missing
--8<---------------cut here---------------end--------------->8---

([omissis] means I'm not copying the exact uuid, sda3 is the first block
device in the BTRFS pool)

All I get now is the guix rescue environment prompt, that I do not know
how to use: I'm not able to boot with BRTFS in degraded mode :-S

Christopher suggested I might be able to at least mount the filesystem
with the degraded option in the guix rescue environment, which might be
something like:

--8<---------------cut here---------------start------------->8---
	 (mkdir "/mnt/broken-root")
	 (mount "/dev/sda3" "/mnt/broken-root" "btrfs" 0 "degraded")
--8<---------------cut here---------------end--------------->8---

but we do not know how to proceed from there.

Obviously I have no way now to reconfigure guix, the only idea I got is
to boot from an USB rescue disk (e.g. grml) and try to do a "btrfs
replace..." from there: that should fix the BTRFS array and should allow
a mount in non-degraded mode, so the next Guix boot should succeed

That machine is physically far away from me and I should collect as much
info as possible before I go there to test for a solution (no remote
serial console unfortunately)

I'm searching the web for a solution, any hint will be greatly
appreciated :-)

Meanwhile milano-guix-1 build machine is offline... :-(

Thank you for your attention, Gio'




[1] AFAIU that is the only way to present a single disk to the OS and
let the OS manage it as part of a **software** RAID pool (hardware RAID
is not an option)

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: cannot boot with BTRFS in degraded mode

[Gábor Boskovits]

Hello Giovanni,

Giovanni Biscuolo <g@xelera.eu> ezt írta (időpont: 2019. szept. 4., Sze,
16:36):

> Hi Guix!
>
> Yesterday I had to physically replace a failed disk on milano-guix-1
> (one of Guix build machines), that disk was part of a BTRFS RAID10
> multi disk array and now the machine is unbootable
>

Sorry to hear that.


> The BTRFS RAID10 array was made of 6 disks and was running well, some
> days ago Christopher Baines found that the 5th disk (/dev/sde) of that
> array failed and was able to remount it in degraded mode in order to
> re-balance the array and go on working without data loss
>
> Unfortunately I was not able to perform a "btrfs replace..."  since
> adding a new disk (we have spare slots) was not detected by the
> kernel... HP ProLiant Smart Array is not so smart after all (aka bye bye
> hot swapping of disks) :-S...
>
> So I had to reboot the server and enter the config tool, added the new
> drive as a new Smart Array logical volume (RAID0 with 1 drive) [1] and
> removed the failed logical volume
>
> The problem now is that the boot process stops when trying to mount the
> BTRFS filesystem, the error is:
>
> --8<---------------cut here---------------start------------->8---
> BTRFS error (device sda3): devid 5 uuid [omissis] is missing
> --8<---------------cut here---------------end--------------->8---
>
> ([omissis] means I'm not copying the exact uuid, sda3 is the first block
> device in the BTRFS pool)
>
> All I get now is the guix rescue environment prompt, that I do not know
> how to use: I'm not able to boot with BRTFS in degraded mode :-S
>
> Christopher suggested I might be able to at least mount the filesystem
> with the degraded option in the guix rescue environment, which might be
> something like:
>
> --8<---------------cut here---------------start------------->8---
>          (mkdir "/mnt/broken-root")
>          (mount "/dev/sda3" "/mnt/broken-root" "btrfs" 0 "degraded")
> --8<---------------cut here---------------end--------------->8---
>
> but we do not know how to proceed from there.
>

I don't know what would work from here, but here are a few ideas:
1. somehow hack the degraded root option into the bootloader config, like
here:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1229456
2. try to switch_root, using /bin/sh as init, and try to fix the bootloader
config from there.
3. see what the original script is doing, either by having a look at how it
is composed:
see for example: gnu/system.scm:
operating-system-default-essential-services,
gnu/services.scm: %boot-service and most prominently:
gnu/services/shepherd.scm: shepherd-boot-gexp

Wdyt?


> Obviously I have no way now to reconfigure guix, the only idea I got is
> to boot from an USB rescue disk (e.g. grml) and try to do a "btrfs
> replace..." from there: that should fix the BTRFS array and should allow
> a mount in non-degraded mode, so the next Guix boot should succeed
>
> That machine is physically far away from me and I should collect as much
> info as possible before I go there to test for a solution (no remote
> serial console unfortunately)
>
> I'm searching the web for a solution, any hint will be greatly
> appreciated :-)
>
> Meanwhile milano-guix-1 build machine is offline... :-(
>
> Thank you for your attention, Gio'
>
>
>
>
> [1] AFAIU that is the only way to present a single disk to the OS and
> let the OS manage it as part of a **software** RAID pool (hardware RAID
> is not an option)
>
> --
> Giovanni Biscuolo
>
> Xelera IT Infrastructures
>

Best regards,
g_bor

-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

Re: cannot boot with BTRFS in degraded mode

[Gábor Boskovits]

I forgot to include one more, that could well be relevant:

Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2019. szept. 4.,
Sze, 22:49):

> Hello Giovanni,
>
> Giovanni Biscuolo <g@xelera.eu> ezt írta (időpont: 2019. szept. 4., Sze,
> 16:36):
>
>> Hi Guix!
>>
>> Yesterday I had to physically replace a failed disk on milano-guix-1
>> (one of Guix build machines), that disk was part of a BTRFS RAID10
>> multi disk array and now the machine is unbootable
>>
>
> Sorry to hear that.
>
>
>> The BTRFS RAID10 array was made of 6 disks and was running well, some
>> days ago Christopher Baines found that the 5th disk (/dev/sde) of that
>> array failed and was able to remount it in degraded mode in order to
>> re-balance the array and go on working without data loss
>>
>> Unfortunately I was not able to perform a "btrfs replace..."  since
>> adding a new disk (we have spare slots) was not detected by the
>> kernel... HP ProLiant Smart Array is not so smart after all (aka bye bye
>> hot swapping of disks) :-S...
>>
>> So I had to reboot the server and enter the config tool, added the new
>> drive as a new Smart Array logical volume (RAID0 with 1 drive) [1] and
>> removed the failed logical volume
>>
>> The problem now is that the boot process stops when trying to mount the
>> BTRFS filesystem, the error is:
>>
>> --8<---------------cut here---------------start------------->8---
>> BTRFS error (device sda3): devid 5 uuid [omissis] is missing
>> --8<---------------cut here---------------end--------------->8---
>>
>> ([omissis] means I'm not copying the exact uuid, sda3 is the first block
>> device in the BTRFS pool)
>>
>> All I get now is the guix rescue environment prompt, that I do not know
>> how to use: I'm not able to boot with BRTFS in degraded mode :-S
>>
>> Christopher suggested I might be able to at least mount the filesystem
>> with the degraded option in the guix rescue environment, which might be
>> something like:
>>
>> --8<---------------cut here---------------start------------->8---
>>          (mkdir "/mnt/broken-root")
>>          (mount "/dev/sda3" "/mnt/broken-root" "btrfs" 0 "degraded")
>> --8<---------------cut here---------------end--------------->8---
>>
>> but we do not know how to proceed from there.
>>
>
> I don't know what would work from here, but here are a few ideas:
> 1. somehow hack the degraded root option into the bootloader config, like
> here:
> https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1229456
> 2. try to switch_root, using /bin/sh as init, and try to fix the
> bootloader config from there.
> 3. see what the original script is doing, either by having a look at how
> it is composed:
> see for example: gnu/system.scm:
> operating-system-default-essential-services,
> gnu/services.scm: %boot-service and most prominently:
> gnu/services/shepherd.scm: shepherd-boot-gexp
>
>

Also take a look at this: gnu/build/linux-boot.scm.


> Wdyt?
>
>
>> Obviously I have no way now to reconfigure guix, the only idea I got is
>> to boot from an USB rescue disk (e.g. grml) and try to do a "btrfs
>> replace..." from there: that should fix the BTRFS array and should allow
>> a mount in non-degraded mode, so the next Guix boot should succeed
>>
>> That machine is physically far away from me and I should collect as much
>> info as possible before I go there to test for a solution (no remote
>> serial console unfortunately)
>>
>> I'm searching the web for a solution, any hint will be greatly
>> appreciated :-)
>>
>> Meanwhile milano-guix-1 build machine is offline... :-(
>>
>> Thank you for your attention, Gio'
>>
>>
>>
>>
>> [1] AFAIU that is the only way to present a single disk to the OS and
>> let the OS manage it as part of a **software** RAID pool (hardware RAID
>> is not an option)
>>
>> --
>> Giovanni Biscuolo
>>
>> Xelera IT Infrastructures
>>
>
> Best regards,
> g_bor
>
> --
> OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21
>

Best regards,
g_bor
-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

Re: cannot boot with BTRFS in degraded mode

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

Hi Gábor,

Gábor Boskovits <boskovits@gmail.com> writes:

[...]

>> Christopher suggested I might be able to at least mount the filesystem
>> with the degraded option in the guix rescue environment, which might be
>> something like:
>>
>> --8<---------------cut here---------------start------------->8---
>>          (mkdir "/mnt/broken-root")
>>          (mount "/dev/sda3" "/mnt/broken-root" "btrfs" 0 "degraded")
>> --8<---------------cut here---------------end--------------->8---
>>
>> but we do not know how to proceed from there.
>
> I don't know what would work from here, but here are a few ideas:
> 1. somehow hack the degraded root option into the bootloader config, like
> here:
> https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1229456

Oh great! I was not able to find what it seems a solution to boot: add
"degraded" to the "rootflags=" kernel parameters (ah my ignorance!)

I also find (just now :-S) this message very helpful:
https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31265.html

> 2. try to switch_root, using /bin/sh as init, and try to fix the bootloader
> config from there.

This is not strictly needed, all I need to boot is entering the grub
editing mode, boot degraded and solve the problem with "btrfs
replace..." and "btrfs balance..."

I was thinking to permanently add "rootflags=[...],degraded" as kernel
parameter but reading the message I mentioned above:

--8<---------------cut here---------------start------------->8---
Altho if it is indeed true that btrfs will now refuse to mount writable 
if it's degraded like that, that's not such a huge issue either, as the 
read-only mount can serve as the same warning.  Still, I certainly prefer 
the refusal to mount entirely without the degraded option, if indeed the 
filesystem is lacking a component device.  There's nothing quite like 
forcing me to actually type in "rootflags=degraded" to rub my face in the 
reality and gravity of the situation I'm in! =:^)
--8<---------------cut here---------------end--------------->8---

I changed my mind: booting in degraded mode should be done "manually"
just to solve the issue :-)

> 3. see what the original script is doing, either by having a look at how it
> is composed:
> see for example: gnu/system.scm:
> operating-system-default-essential-services,
> gnu/services.scm: %boot-service and most prominently:
> gnu/services/shepherd.scm: shepherd-boot-gexp
>
> Wdyt?

I'm certailny going to study how Guix instruments the booting of a
machine, but I need time... and I'm late :-S

Thank you very much for the pointers, indeed!

[...]

Ciao, Gio'.

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: cannot boot with BTRFS in degraded mode

[Maxim Cournoyer]

Hello Giovanni!

Giovanni Biscuolo <g@xelera.eu> writes:

[...]

> [...] all I need to boot is entering the grub editing mode, boot
> degraded and solve the problem with "btrfs replace..." and "btrfs
> balance..."
>
> I was thinking to permanently add "rootflags=[...],degraded" as kernel
> parameter but reading the message I mentioned above:
>
> Altho if it is indeed true that btrfs will now refuse to mount writable 
> if it's degraded like that, that's not such a huge issue either, as the 
> read-only mount can serve as the same warning.  Still, I certainly prefer 
> the refusal to mount entirely without the degraded option, if indeed the 
> filesystem is lacking a component device.  There's nothing quite like 
> forcing me to actually type in "rootflags=degraded" to rub my face in the 
> reality and gravity of the situation I'm in! =:^)
>
> I changed my mind: booting in degraded mode should be done "manually"
> just to solve the issue :-)

Also note that for the time being, the init script of Guix doesn't honor
the "rootflags" kernel argument.

The good news is that there is a patch awaiting testing/review that adds
support for it here: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37305.

Cheers,

Maxim

Re: cannot boot with BTRFS in degraded mode

[Giovanni Biscuolo]

Hello Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

> Also note that for the time being, the init script of Guix doesn't honor
> the "rootflags" kernel argument.

AFAIU this means Guix will not configure the grub parameter in the grub
config, but if I pass "rootflags" via grub "manually" that should be
passed to the kernel: right?

Anyway, with the help of a friend, in a few hours we are going to test
this solution :-)

> The good news is that there is a patch awaiting testing/review that adds
> support for it here:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37305.

AFAIU it's a good patch, thanks!

IMHO actually it's two patches merged togheter: "honor rootflags kernel
parameter" and "Allow booting from a Btrfs subvolume", anyway it's a big
imrovement for BTRFS users

I'll try them and will report back!

Thanks, Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

Re: cannot boot with BTRFS in degraded mode

[Maxim Cournoyer]

Giovanni Biscuolo <g@xelera.eu> writes:

> Hello Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>> Also note that for the time being, the init script of Guix doesn't honor
>> the "rootflags" kernel argument.
>
> AFAIU this means Guix will not configure the grub parameter in the grub
> config, but if I pass "rootflags" via grub "manually" that should be
> passed to the kernel: right?

Fixing the Grub config (manually or not) is one thing, but not enough,
as it is the init script (contained within the initrd) run by the kernel
that mounts the root partition.  This script currently doesn't honor the
rootflags= argument given to the Linux kernel command line, which means
that you cannot pass extra options (such as 'degraded') to mount your
root partition during the early boot process of the system.

> Anyway, with the help of a friend, in a few hours we are going to test
> this solution :-)

Good luck!

>> The good news is that there is a patch awaiting testing/review that adds
>> support for it here:
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37305.
>
> AFAIU it's a good patch, thanks!
>
> IMHO actually it's two patches merged togheter: "honor rootflags kernel
> parameter" and "Allow booting from a Btrfs subvolume", anyway it's a big
> imrovement for BTRFS users

Yes, there are different commits (patches) addressing those
sub-problems.  The bigger objective was to be able to boot from a Btrfs
subvolume.

> I'll try them and will report back!
>
> Thanks, Gio'

I will look forward to the report!

Maxim

Re: cannot boot with BTRFS in degraded mode

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 706 bytes --]


Giovanni Biscuolo <g@xelera.eu> writes:

> Hello Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>> Also note that for the time being, the init script of Guix doesn't honor
>> the "rootflags" kernel argument.
>
> AFAIU this means Guix will not configure the grub parameter in the grub
> config, but if I pass "rootflags" via grub "manually" that should be
> passed to the kernel: right?

Unfortunately, I think this means that without the patch posted, the
rootflags parameter won't be used.

The flag isn't indended to be passed to Linux, but to the code managed
by Guix that mounts the root filesystem (which is what the patch changes
to take note of the rootflags argument).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: cannot boot with BTRFS in degraded mode

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]

Hello Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

> Fixing the Grub config (manually or not) is one thing, but not enough,
> as it is the init script (contained within the initrd)

Yes I was obviously wrong: the initrd must be updated, also!

> run by the kernel
> that mounts the root partition.  This script currently doesn't honor the
> rootflags= argument given to the Linux kernel command line, which means
> that you cannot pass extra options (such as 'degraded') to mount your
> root partition during the early boot process of the system.

Yes, now I understand, thanks!

>> Anyway, with the help of a friend, in a few hours we are going to test
>> this solution :-)
>
> Good luck!

The only viable solution to this problem is to boot a rescue system and
fix the BTRFS array there: I'm doing it now via grml (remotely via ssh
with the help of a friend who "manually" started the network and ssh
daemon)

For the records, this is what I did:

1. btrfs replace start 5 /dev/sde /target (since the missimg one was the
5th)

2. btrfs replace status /target

it took less then 10 mins to replace a 1TB disk with 76.28GiB used:

--8<---------------cut here---------------start------------->8---
Started on  6.Sep 09:21:44, finished on  6.Sep 09:40:45, 0 write errs, 0 uncorr. read errs
--8<---------------cut here---------------end--------------->8---

[...]

Thank you again for your support! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: cannot boot with BTRFS in degraded mode

[Maxim Cournoyer]

Hello Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:

> Hello Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>> Fixing the Grub config (manually or not) is one thing, but not enough,
>> as it is the init script (contained within the initrd)
>
> Yes I was obviously wrong: the initrd must be updated, also!
>
>> run by the kernel
>> that mounts the root partition.  This script currently doesn't honor the
>> rootflags= argument given to the Linux kernel command line, which means
>> that you cannot pass extra options (such as 'degraded') to mount your
>> root partition during the early boot process of the system.
>
> Yes, now I understand, thanks!
>
>>> Anyway, with the help of a friend, in a few hours we are going to test
>>> this solution :-)
>>
>> Good luck!
>
> The only viable solution to this problem is to boot a rescue system and
> fix the BTRFS array there: I'm doing it now via grml (remotely via ssh
> with the help of a friend who "manually" started the network and ssh
> daemon)
>
> For the records, this is what I did:
>
> 1. btrfs replace start 5 /dev/sde /target (since the missimg one was the
> 5th)
>
> 2. btrfs replace status /target
>
> it took less then 10 mins to replace a 1TB disk with 76.28GiB used:
>
> Started on  6.Sep 09:21:44, finished on  6.Sep 09:40:45, 0 write errs, 0 uncorr. read errs

Great! It looks like you're all set! Thank you for maintaining a build
machine :-).

Maxim