From: "Gábor Boskovits" <boskovits@gmail.com>
To: Giovanni Biscuolo <g@xelera.eu>
Cc: help-guix <help-guix@gnu.org>
Subject: Re: cannot boot with BTRFS in degraded mode
Date: Wed, 4 Sep 2019 22:49:09 +0200 [thread overview]
Message-ID: <CAE4v=pigS23fM9Y380i3zGQT-_+nNsWzueBpbXn7Pn6PoKdDrA@mail.gmail.com> (raw)
In-Reply-To: <87lfv4163w.fsf@roquette.mug.biscuolo.net>
Hello Giovanni,
Giovanni Biscuolo <g@xelera.eu> ezt írta (időpont: 2019. szept. 4., Sze,
16:36):
> Hi Guix!
>
> Yesterday I had to physically replace a failed disk on milano-guix-1
> (one of Guix build machines), that disk was part of a BTRFS RAID10
> multi disk array and now the machine is unbootable
>
Sorry to hear that.
> The BTRFS RAID10 array was made of 6 disks and was running well, some
> days ago Christopher Baines found that the 5th disk (/dev/sde) of that
> array failed and was able to remount it in degraded mode in order to
> re-balance the array and go on working without data loss
>
> Unfortunately I was not able to perform a "btrfs replace..." since
> adding a new disk (we have spare slots) was not detected by the
> kernel... HP ProLiant Smart Array is not so smart after all (aka bye bye
> hot swapping of disks) :-S...
>
> So I had to reboot the server and enter the config tool, added the new
> drive as a new Smart Array logical volume (RAID0 with 1 drive) [1] and
> removed the failed logical volume
>
> The problem now is that the boot process stops when trying to mount the
> BTRFS filesystem, the error is:
>
> --8<---------------cut here---------------start------------->8---
> BTRFS error (device sda3): devid 5 uuid [omissis] is missing
> --8<---------------cut here---------------end--------------->8---
>
> ([omissis] means I'm not copying the exact uuid, sda3 is the first block
> device in the BTRFS pool)
>
> All I get now is the guix rescue environment prompt, that I do not know
> how to use: I'm not able to boot with BRTFS in degraded mode :-S
>
> Christopher suggested I might be able to at least mount the filesystem
> with the degraded option in the guix rescue environment, which might be
> something like:
>
> --8<---------------cut here---------------start------------->8---
> (mkdir "/mnt/broken-root")
> (mount "/dev/sda3" "/mnt/broken-root" "btrfs" 0 "degraded")
> --8<---------------cut here---------------end--------------->8---
>
> but we do not know how to proceed from there.
>
I don't know what would work from here, but here are a few ideas:
1. somehow hack the degraded root option into the bootloader config, like
here:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1229456
2. try to switch_root, using /bin/sh as init, and try to fix the bootloader
config from there.
3. see what the original script is doing, either by having a look at how it
is composed:
see for example: gnu/system.scm:
operating-system-default-essential-services,
gnu/services.scm: %boot-service and most prominently:
gnu/services/shepherd.scm: shepherd-boot-gexp
Wdyt?
> Obviously I have no way now to reconfigure guix, the only idea I got is
> to boot from an USB rescue disk (e.g. grml) and try to do a "btrfs
> replace..." from there: that should fix the BTRFS array and should allow
> a mount in non-degraded mode, so the next Guix boot should succeed
>
> That machine is physically far away from me and I should collect as much
> info as possible before I go there to test for a solution (no remote
> serial console unfortunately)
>
> I'm searching the web for a solution, any hint will be greatly
> appreciated :-)
>
> Meanwhile milano-guix-1 build machine is offline... :-(
>
> Thank you for your attention, Gio'
>
>
>
>
> [1] AFAIU that is the only way to present a single disk to the OS and
> let the OS manage it as part of a **software** RAID pool (hardware RAID
> is not an option)
>
> --
> Giovanni Biscuolo
>
> Xelera IT Infrastructures
>
Best regards,
g_bor
--
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21
next prev parent reply other threads:[~2019-09-04 20:49 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-04 14:36 cannot boot with BTRFS in degraded mode Giovanni Biscuolo
2019-09-04 20:49 ` Gábor Boskovits [this message]
2019-09-04 21:02 ` Gábor Boskovits
2019-09-05 14:17 ` Giovanni Biscuolo
2019-09-06 1:00 ` Maxim Cournoyer
2019-09-06 6:30 ` Giovanni Biscuolo
2019-09-06 7:20 ` Maxim Cournoyer
2019-09-06 9:47 ` Giovanni Biscuolo
2019-09-07 0:20 ` Maxim Cournoyer
2019-09-06 7:21 ` Christopher Baines
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAE4v=pigS23fM9Y380i3zGQT-_+nNsWzueBpbXn7Pn6PoKdDrA@mail.gmail.com' \
--to=boskovits@gmail.com \
--cc=g@xelera.eu \
--cc=help-guix@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.