bug#38086: RAID installation script with ‘mdadm’ no longer works

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Ludovic Courtès]

Hello,

Looks like our RAID installation method no longer works, as can be seen
at <https://ci.guix.gnu.org/build/1906208/details>:

--8<---------------cut here---------------start------------->8---
+ guix --version
guix (GNU Guix) c4de60ac3c6aa5b46519011af89988215c347e9e
Copyright (C) 2019 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ export GUIX_BUILD_OPTIONS=--no-grafts
+ GUIX_BUILD_OPTIONS=--no-grafts
+ parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart primary ext2 3M 600M mkpart primary ext2 600M 1200M set 1 boot on set 1 bios_grub on
+ mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 /dev/vdb2 /dev/vdb3
mdadm: chunk size defaults to 512K
mdadm: Defaulting to version 1.2 metadata
[   13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with default_layout setting
[   13.894691] md/raid0: please set raid0.default_layout to 1 or 2
[   13.896000] md: pers->run() failed ...
mdadm: RUN_ARRAY failed: Unknown error 524
[   13.901603] md: md0 stopped.
--8<---------------cut here---------------end--------------->8---

Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?

We should then update (gnu tests install) and the manual accordingly.

Thanks,
Ludo’.

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1981 bytes --]

Hello Ludo,


Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2019. nov. 6., Sze,
11:14):

> Hello,
>
> Looks like our RAID installation method no longer works, as can be seen
> at <https://ci.guix.gnu.org/build/1906208/details>:
>
> --8<---------------cut here---------------start------------->8---
> + guix --version
> guix (GNU Guix) c4de60ac3c6aa5b46519011af89988215c347e9e
> Copyright (C) 2019 the Guix authors
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> + export GUIX_BUILD_OPTIONS=--no-grafts
> + GUIX_BUILD_OPTIONS=--no-grafts
> + parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart
> primary ext2 3M 600M mkpart primary ext2 600M 1200M set 1 boot on set 1
> bios_grub on
> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2
> /dev/vdb2 /dev/vdb3
> mdadm: chunk size defaults to 512K
> mdadm: Defaulting to version 1.2 metadata
> [   13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with
> default_layout setting
> [   13.894691] md/raid0: please set raid0.default_layout to 1 or 2
> [   13.896000] md: pers->run() failed ...
> mdadm: RUN_ARRAY failed: Unknown error 524
> [   13.901603] md: md0 stopped.
> --8<---------------cut here---------------end--------------->8---
>
> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
>

On kernel 5.3.4 and above the
raid0.default_layout=2 kernel boot paramter should be set. We should
generate our grub configuration accordingly.

See this for reference:
https://blog.icod.de/2019/10/10/caution-kernel-5-3-4-and-raid0-default_layout/



> We should then update (gnu tests install) and the manual accordingly.
>
> Thanks,
> Ludo’.
>
Best regards,
g_bor

-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

[-- Attachment #2: Type: text/html, Size: 3523 bytes --]

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

Hi Gábor,

Gábor Boskovits <boskovits@gmail.com> skribis:

>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2
>> /dev/vdb2 /dev/vdb3
>> mdadm: chunk size defaults to 512K
>> mdadm: Defaulting to version 1.2 metadata
>> [   13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with
>> default_layout setting
>> [   13.894691] md/raid0: please set raid0.default_layout to 1 or 2
>> [   13.896000] md: pers->run() failed ...
>> mdadm: RUN_ARRAY failed: Unknown error 524
>> [   13.901603] md: md0 stopped.
>> --8<---------------cut here---------------end--------------->8---
>>
>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
>>
>
> On kernel 5.3.4 and above the
> raid0.default_layout=2 kernel boot paramter should be set. We should
> generate our grub configuration accordingly.

That’s part of the solution, thank you!

With the patch below, the “raid-root-os” test successfully installs the
system to a RAID0 device, but then that system fails to boot with:

--8<---------------cut here---------------start------------->8---
Booting from Hard Disk...
GRUB loading.
Welcome to GRUB!

error: invalid arch-independent ELF magic.
Entering rescue mode...
--8<---------------cut here---------------end--------------->8---

(It sits there forever.)

Are we missing something in ‘grub.cfg’?  If so, I wonder if that problem
arose with the upgrade in commit
069ab3bbfde704760acaca20dff8a29d167c6be5.

Thoughts?

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1072 bytes --]

diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 22c9554705..5e421f7c54 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -543,7 +543,8 @@ where /gnu lives on a separate partition.")
     (bootloader (bootloader-configuration
                  (bootloader grub-bootloader)
                  (target "/dev/vdb")))
-    (kernel-arguments '("console=ttyS0"))
+    (kernel-arguments '("console=ttyS0"
+                        "raid0.default_layout=2"))
 
     ;; Add a kernel module for RAID-0 (aka. "stripe").
     (initrd-modules (cons "raid0" %base-initrd-modules))
@@ -578,9 +579,11 @@ export GUIX_BUILD_OPTIONS=--no-grafts
 parted --script /dev/vdb mklabel gpt \\
   mkpart primary ext2 1M 3M \\
   mkpart primary ext2 3M 600M \\
-  mkpart primary ext2 600M 1200M \\
+  mkpart primary ext2 600M 1.4G \\
   set 1 boot on \\
   set 1 bios_grub on
+modprobe raid0
+echo 1 > /sys/module/raid0/parameters/default_layout
 mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 \\
   /dev/vdb2 /dev/vdb3
 mkfs.ext4 -L root-fs /dev/md0

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Ludovic Courtès]

An update: this is the last known good test:

  https://berlin.guixsd.org/build/1793057/details

and this is the first known-bad (‘mdadm’ failing with “cannot assemble
multi-zone RAID0 without default_layout setting”):

  https://berlin.guixsd.org/build/1795351/details

We have to resort to an ungly hack to get the evaluation number and
corresponding commit of each build because they aren’t accessible over
HTTP (which is unfortunate!):

--8<---------------cut here---------------start------------->8---
sqlite> select * from builds where rowid = 1793057;
/gnu/store/618hm2w0clcrxz16yww846mgqdc1l4s0-raid-root-os.drv|7863|test.raid-root-os.i686-linux|i686-linux|raid-root-os||0|1570439988|1570459635|1570459744
sqlite> select * from checkouts where evaluation = 7863;
guix-master|7b6a8e23b0de18262a42e44432f955517d71d796|7863|guix|/gnu/store/7sd2lwj83n6kyn66p9bdgs5yvzqnl539-guix-7b6a8e2
sqlite> select * from builds where rowid = 1795351;
/gnu/store/qskl45gw9y9hd8qp7s5451d53pvpc60q-raid-root-os.drv|7867|test.raid-root-os.i686-linux|i686-linux|raid-root-os||2|1570440409|0|1570457622
sqlite> select * from checkouts where evaluation = 7867;
guix-master|7d82e920717f08bceb42bb570d786dff233171e1|7867|guix|/gnu/store/b2cq9zhdsz4qri2xkg3rgwmyri0wyxxb-guix-7d82e92
--8<---------------cut here---------------end--------------->8---

So the commit that introduced the change of behavior of ‘mdadm’ is:

--8<---------------cut here---------------start------------->8---
commit 7d82e920717f08bceb42bb570d786dff233171e1
Date:   Sun Oct 6 06:07:15 2019 +0000

    gnu: linux-libre: Update to 5.3.4.
--8<---------------cut here---------------end--------------->8---

And indeed that brings us back to:

  https://blog.icod.de/2019/10/10/caution-kernel-5-3-4-and-raid0-default_layout/

Hmm alright, nothing new here.  Oh well!

Ludo’.

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1535 bytes --]

On 2019-11-12, Ludovic Courtès wrote:
> Gábor Boskovits <boskovits@gmail.com> skribis:
>
>>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2
>>> /dev/vdb2 /dev/vdb3
>>> mdadm: chunk size defaults to 512K
>>> mdadm: Defaulting to version 1.2 metadata
>>> [   13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with
>>> default_layout setting
>>> [   13.894691] md/raid0: please set raid0.default_layout to 1 or 2
>>> [   13.896000] md: pers->run() failed ...
>>> mdadm: RUN_ARRAY failed: Unknown error 524
>>> [   13.901603] md: md0 stopped.
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
>>>
>>
>> On kernel 5.3.4 and above the
>> raid0.default_layout=2 kernel boot paramter should be set. We should
>> generate our grub configuration accordingly.

So, this might be sort of a tangent, but I'm wondering why you're
testing raid0 (striping, for performance+capacity at risk of data loss)
instead of raid1 (mirroring, for redundancy, fast reads, slow writes,
half capacity of storage), or another raid level with more disks (raid5,
raid6, raid10). raid1 would be the simplest to switch the code to, since
it uses only two disks.


The issue triggering this bug might be a non-issue on other raid levels
that in my mind might make more sense for rootfs. Or maybe people have
use-casese for rootfs on raid0 that I'm too uncreative to think of? :)


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1788 bytes --]

Vagrant Cascadian <vagrant@debian.org> ezt írta (időpont: 2020. jan. 17.,
Pén 23:42):

> On 2019-11-12, Ludovic Courtès wrote:
> > Gábor Boskovits <boskovits@gmail.com> skribis:
> >
> >>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2
> >>> /dev/vdb2 /dev/vdb3
> >>> mdadm: chunk size defaults to 512K
> >>> mdadm: Defaulting to version 1.2 metadata
> >>> [   13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with
> >>> default_layout setting
> >>> [   13.894691] md/raid0: please set raid0.default_layout to 1 or 2
> >>> [   13.896000] md: pers->run() failed ...
> >>> mdadm: RUN_ARRAY failed: Unknown error 524
> >>> [   13.901603] md: md0 stopped.
> >>> --8<---------------cut here---------------end--------------->8---
> >>>
> >>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
> >>>
> >>
> >> On kernel 5.3.4 and above the
> >> raid0.default_layout=2 kernel boot paramter should be set. We should
> >> generate our grub configuration accordingly.
>
> So, this might be sort of a tangent, but I'm wondering why you're
> testing raid0 (striping, for performance+capacity at risk of data loss)
> instead of raid1 (mirroring, for redundancy, fast reads, slow writes,
> half capacity of storage), or another raid level with more disks (raid5,
> raid6, raid10). raid1 would be the simplest to switch the code to, since
> it uses only two disks.
>
>
> The issue triggering this bug might be a non-issue on other raid levels
> that in my mind might make more sense for rootfs. Or maybe people have
> use-casese for rootfs on raid0 that I'm too uncreative to think of? :)
>

I often see raid 10 as root. I believe it might make sense to test that
setup.

>
>
> live well,
>   vagrant
>

[-- Attachment #2: Type: text/html, Size: 2610 bytes --]

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

Hi!

Vagrant Cascadian <vagrant@debian.org> skribis:

> So, this might be sort of a tangent, but I'm wondering why you're
> testing raid0 (striping, for performance+capacity at risk of data loss)
> instead of raid1 (mirroring, for redundancy, fast reads, slow writes,
> half capacity of storage), or another raid level with more disks (raid5,
> raid6, raid10). raid1 would be the simplest to switch the code to, since
> it uses only two disks.

Good point!  I guess it would make sense to test RAID1, indeed.

I gave it a shot with the patch below.  Problem is that installation
seemingly hangs here:

--8<---------------cut here---------------start------------->8---
+ parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart primary ext2 3M 1.4G mkpart primary ext2 1.4G 2.8G set 1 boot on set 1 bios_grub on
+ mdadm --create /dev/md0 --verbose --level=mirror --raid-devices=2 /dev/vdb2 /dev/vdb3
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: size set to 1361920K
mdadm: largest drive (/dev/vdb3) exceeds size (1361920K) by more than 1%
--8<---------------cut here---------------end--------------->8---

As you can see, it’s attempting to make a RAID1 device out of two
partitions (not two disks), which makes no sense in the real world, but
is easier to handle here.  So I wonder if this is what’s causing it to
hang…

Thoughts?

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1746 bytes --]

diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 8842d48df8..12e6eb26df 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -546,8 +546,8 @@ where /gnu lives on a separate partition.")
                  (target "/dev/vdb")))
     (kernel-arguments '("console=ttyS0"))
 
-    ;; Add a kernel module for RAID-0 (aka. "stripe").
-    (initrd-modules (cons "raid0" %base-initrd-modules))
+    ;; Add a kernel module for RAID-1 (aka. "mirror").
+    (initrd-modules (cons "raid1" %base-initrd-modules))
 
     (mapped-devices (list (mapped-device
                            (source (list "/dev/vda2" "/dev/vda3"))
@@ -578,11 +578,11 @@ guix --version
 export GUIX_BUILD_OPTIONS=--no-grafts
 parted --script /dev/vdb mklabel gpt \\
   mkpart primary ext2 1M 3M \\
-  mkpart primary ext2 3M 600M \\
-  mkpart primary ext2 600M 1200M \\
+  mkpart primary ext2 3M 1.4G \\
+  mkpart primary ext2 1.4G 2.8G \\
   set 1 boot on \\
   set 1 bios_grub on
-mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 \\
+mdadm --create /dev/md0 --verbose --level=mirror --raid-devices=2 \\
   /dev/vdb2 /dev/vdb3
 mkfs.ext4 -L root-fs /dev/md0
 mount /dev/md0 /mnt
@@ -605,7 +605,7 @@ by 'mdadm'.")
                                                %raid-root-os-source
                                                #:script
                                                %raid-root-installation-script
-                                               #:target-size (* 1300 MiB)))
+                                               #:target-size (* 2800 MiB)))
                          (command (qemu-command/writable-image image)))
       (run-basic-test %raid-root-os
                       `(,@command) "raid-root-os")))))

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Ludovic Courtès 写道：
> As you can see, it’s attempting to make a RAID1 device out of 
> two
> partitions (not two disks), which makes no sense in the real 
> world, but
> is easier to handle here.  So I wonder if this is what’s causing 
> it to
> hang…

It's just waiting for input:

  $ # dd & losetup magic, where loop0 is 20% larger than loop1
  $ sudo mdadm --create /dev/md0 --verbose --level=mirror 
  --raid-devices=2 /dev/loop{0,1}
  mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
  mdadm: size set to 101376K
  mdadm: largest drive (/dev/loop1) exceeds size (101376K) by more 
  than 1%
  Continue creating array?

Adding --force does not avoid this.

I recommend tweaking the partition table to make both members 
equal, but a ‘yes|’ also works if you're in a hurry ;-)

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Ludovic Courtès]

Hi Tobias!

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> It's just waiting for input:
>
>  $ # dd & losetup magic, where loop0 is 20% larger than loop1
>  $ sudo mdadm --create /dev/md0 --verbose --level=mirror
> --raid-devices=2 /dev/loop{0,1}
>  mdadm: Note: this array has metadata at the start and
>    may not be suitable as a boot device.  If you plan to
>    store '/boot' on this device please ensure that
>    your boot-loader understands md/v1.x metadata, or use
>    --metadata=0.90
>  mdadm: size set to 101376K
>  mdadm: largest drive (/dev/loop1) exceeds size (101376K) by more than
> 1%
>  Continue creating array?

D’oh, I hadn’t seen that message.

> Adding --force does not avoid this.
>
> I recommend tweaking the partition table to make both members equal,
> but a ‘yes|’ also works if you're in a hurry ;-)

“yes|” works like a charm, I went this that.

Pushed in commit 3adf320e44e54017a67f219ce9667a379c393dad, thank you!

Ludo’.

bug#38086: RAID installation script with ‘mdadm’ no longer works

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 373 bytes --]

Ludo',

Ludovic Courtès 写道：
>>  Continue creating array?
>
> D’oh, I hadn’t seen that message.

I doubt it was there for you to see in Guix's output.  Things not 
ending with a newline tend to get lost easily due to 
line-buffering.  Probably not worth worrying about.

> “yes|” works like a charm, I went this that.

‘Beautiful’,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]