bug#65177: udevd error with lvm-raid array leading to race condition with luks

bug#65177: udevd error with lvm-raid array leading to race condition with luks

[Adrien 'neox' Bourmault]

Hi there.
My setup is the following (LVM array containing a LUKS partition):

   (mapped-devices
    (list
     (mapped-device
       (source "HOMERAID")
       (target "HOMERAID-HOMERAID_data")
       (type lvm-device-mapping))
     (mapped-device
       (source "/dev/mapper/HOMERAID-HOMERAID_data")
       (target "luks-f0a72a6c-499b-4445-8d13-21dc99337752")
       (type luks-device-mapping))))

   (file-systems
     (cons*
       (file-system
         (mount-point "/")
         (device (uuid "2e44f3f7-bb6b-43ac-933a-e8992bf10d29" 'ext4))
         (type "ext4"))
       (file-system
         (mount-point "/home")
         (device "/dev/mapper/luks-f0a72a6c-499b-4445-8d13-21dc99337752")
         (type "ext4")
         (dependencies mapped-devices))
       (file-system
         (mount-point "/boot/efi")
         (device (uuid "DC58-946E" 'fat32))
         (type "vfat"))
      %base-file-systems)))

I use Guix System since 2022, and never had any problem booting with 
this configuration. But I did update my system (and reconfigure it) last 
week and now I can't boot. I don't have any older generation to restore 
(yes I'm dumb, I executed a delete-generations to show how it works to a 
friend), but I think the last working one was from July, 25th or just a 
bit older.

I have in my /var/log/messages :

   Aug  9 11:40:27 localhost vmunix: [    7.525877] udevd[515]: failed 
to execute '/usr/bin/systemd-run' '/usr/bin/systemd-run --no-block 
--property DefaultDependencies=no --unit lvm-activate-HOMERAID 
/gnu/store/hffkn63zx2zjadawrkxpnr486frc9n74-lvm2-2.03.21/sbin/lvm 
vgchange -aay --autoactivation event HOMERAID': No such file or directory

On the screen, during boot, I obtain a wall of messages like (not the 
exact message, sorry, can't find on syslog) "Device 
HOMERAID-HOMERAID_data could not be found: does not exist or access 
denied" and after that the system hangs.

However, I can boot successfully if and only if I press Scroll Lock just 
after modesetting and before this message ("Device ... not be found") 
appears and when I press it again (to deactivate its effect) after some 
seconds, the LUKS passphrase prompt appears and allows me to boot properly.

I can use my system but it requires multiple tries each time x) I have 
to be really synchronized with the modesetting and the message xD

Thank you very much.

Freely,
-- 
Adrien Bourmault
Co-maintainer, GNU Boot project
Elected member, XMPP Standards Foundation
Associate member, Free Software Foundation
Trésorier, Association Libre en Communs (https://www.a-lec.org)
GPG : 1DF1132CF1658A8559025C98AAD6B069819E6979

bug#65177: udevd error with lvm-raid array leading to race condition with luks

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

Hi Adrien,

Adrien 'neox' Bourmault <neox@gnu.org> writes:

>    Aug  9 11:40:27 localhost vmunix: [    7.525877] udevd[515]: failed 
> to execute '/usr/bin/systemd-run' '/usr/bin/systemd-run --no-block 
> --property DefaultDependencies=no --unit lvm-activate-HOMERAID 
> /gnu/store/hffkn63zx2zjadawrkxpnr486frc9n74-lvm2-2.03.21/sbin/lvm 
> vgchange -aay --autoactivation event HOMERAID': No such file or directory

Since lvm2 2.03.14, the included udev rules use systemd-run to run
vgchange and activate the volume group.  lvm2 was updated recently from
2.03.11 to 2.03.21, then 2.03.22 (cc'ing tobias), and probably started
exhibiting this behavior then.  I think we can probably remove the
indirection through systemd-run and directly run vgchange.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#65177: udevd error with lvm-raid array leading to race condition with luks

[Yann Dupont]

hello everyone, we're also victims of this bug, in an even simpler use case.

[…]
(file-system
                                 (device "/dev/mapper/VG0-DATA")
                                 (mount-point "/VG0-DATA")
                                 (type "ext4"))
[…]

The culprit seems to be 69-dm-lvm.rules

[ 18.226226] udevd[115]: failed to execute '/usr/bin/systemd-run' 
'/usr/bin/systemd-run --no-block --property DefaultDependencies=no 
--unit lvm-activate-VG0 
/gnu/store/0hndg947ywdl5izvy63ny38hyywci66k-lvm2-2.03.22/sbin/lvm vy

I can confirm that when using time-machine to revert to lvm2 2-03.11 
versions, the VM boots.

cheers,

bug#65177: udevd error with lvm-raid array leading to race condition with luks

[Yann Dupont]

[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]

Hi, as suggested by Josselin, I tested the following patch and it seems 
to do the job here.


Be careful, I'm not an udev or lvm2 specialist at all and basically, I 
don't really know if what I did is the right way to do it.


All I can say is that the VMs now boot.


Cheers,



diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 91109c41d9..28b3c1e0bf 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -4421,6 +4421,7 @@ (define-public lvm2
                (sha256
                 (base32
"0z6w6bknhwh1n3qfkb5ij6x57q3wjf28lq3l8kh7rkhsplinjnjc"))
+              (patches (search-patches "lvm2-no-systemd.patch"))
                (modules '((guix build utils)))
                (snippet
                 '(begin
diff --git a/gnu/packages/patches/lvm2-no-systemd.patch 
b/gnu/packages/patches/lvm2-no-systemd.patch
new file mode 100644
index 0000000000..7e8a37abcc
--- /dev/null
+++ b/gnu/packages/patches/lvm2-no-systemd.patch
@@ -0,0 +1,13 @@
+diff --git a/udev/69-dm-lvm.rules.in b/udev/69-dm-lvm.rules.in
+index ff1568145..8879a2ef9 100644
+--- a/udev/69-dm-lvm.rules.in
++++ b/udev/69-dm-lvm.rules.in
+@@ -76,7 +76,7 @@ LABEL="lvm_scan"
+ # it's better suited to appearing in the journal.
+
+ IMPORT{program}="(LVM_EXEC)/lvm pvscan --cache --listvg 
--checkcomplete --vgonline --autoactivation event --udevoutput 
--journal=output $env{DEVNAME}"
+-ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(SYSTEMDRUN) --no-block 
--property DefaultDependencies=no --unit 
lvm-activate-$env{LVM_VG_NAME_COMPLETE} (LVM_EXEC)/lvm vgchange -aay 
--autoactivation event $env{LVM_VG_NAME_COMPLETE}"
++ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(LVM_EXEC)/lvm vgchange -aay 
--autoactivation event $env{LVM_VG_NAME_COMPLETE}"
+ GOTO="lvm_end"
+
+ LABEL="lvm_end"


[-- Attachment #2: Type: text/html, Size: 4177 bytes --]

bug#65177: udevd error with lvm-raid array leading to race condition with luks

[Simon Tournier]

Hi,

On Thu, 14 Sep 2023 at 18:23, Yann Dupont <Yann.Dupont@univ-nantes.fr> wrote:

> All I can say is that the VMs now boot.

WDYT about this patch?

> diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
> index 91109c41d9..28b3c1e0bf 100644
> --- a/gnu/packages/linux.scm
> +++ b/gnu/packages/linux.scm
> @@ -4421,6 +4421,7 @@ (define-public lvm2
>                (sha256
>                 (base32
>                  "0z6w6bknhwh1n3qfkb5ij6x57q3wjf28lq3l8kh7rkhsplinjnjc"))
> +              (patches (search-patches "lvm2-no-systemd.patch"))
>                (modules '((guix build utils)))
>                (snippet
>                 '(begin
> diff --git a/gnu/packages/patches/lvm2-no-systemd.patch
> b/gnu/packages/patches/lvm2-no-systemd.patch
> new file mode 100644
> index 0000000000..7e8a37abcc
> --- /dev/null
> +++ b/gnu/packages/patches/lvm2-no-systemd.patch
> @@ -0,0 +1,13 @@
> +diff --git a/udev/69-dm-lvm.rules.in b/udev/69-dm-lvm.rules.in
> +index ff1568145..8879a2ef9 100644
> +--- a/udev/69-dm-lvm.rules.in
> ++++ b/udev/69-dm-lvm.rules.in
> +@@ -76,7 +76,7 @@ LABEL="lvm_scan"
> + # it's better suited to appearing in the journal.
> +
> + IMPORT{program}="(LVM_EXEC)/lvm pvscan --cache --listvg --checkcomplete
> --vgonline --autoactivation event --udevoutput --journal=output $env{DEVNAME}"
> +-ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(SYSTEMDRUN) --no-block
> --property DefaultDependencies=no --unit lvm-activate-$env
> {LVM_VG_NAME_COMPLETE} (LVM_EXEC)/lvm vgchange -aay --autoactivation event
> $env{LVM_VG_NAME_COMPLETE}"
> ++ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(LVM_EXEC)/lvm vgchange -aay
> --autoactivation event $env{LVM_VG_NAME_COMPLETE}"
> + GOTO="lvm_end"
> +
> + LABEL="lvm_end"

Josselin, is it what you had in mind?

Cheers,
simon

bug#65177: udevd error with lvm-raid array leading to race condition with luks

[Simon Tournier]

Hi,

On Mon, 25 Sep 2023 at 09:35, Simon Tournier <zimon.toutoune@gmail.com> wrote:

>> All I can say is that the VMs now boot.
>
> WDYT about this patch?

For easing the discussion, I extracted Yann’s diff and prepared a patch
ready to merge.  See:

    https://issues.guix.gnu.org/issue/66586

Cheers,
simon

bug#65177: udevd error with lvm-raid array leading to race condition with luks

[Simon Tournier]

Hi,

>>> All I can say is that the VMs now boot.
>>
>> WDYT about this patch?
>
> For easing the discussion, I extracted Yann’s diff and prepared a patch
> ready to merge.  See:
>
>     https://issues.guix.gnu.org/issue/66586

Patch pushed as c0895371c5759c7d9edb330774e90f192cc4cf2c.

Closing.

Feel free to reopen if the patch does not fix the issue.

Cheers,
simon

bug#65177: udevd error with lvm-raid array leading to race

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

Hi,

I think the patch is wrong.  No comment regarding the udev rule itself (I do not know much about it), it does spam the logs with:

Oct 31 18:31:35 localhost vmunix: [   60.013538] udevd[327]: invalid key/value pair in file /gnu/store/a3xvv1zxzyb53r5r752q8nh9v9vd150h-udev-rules/69-dm-lvm.rules on line 78, starting at character 1 ('I')
Oct 31 18:31:35 localhost vmunix: [   60.013544] udevd[327]: invalid key/value pair in file /gnu/store/a3xvv1zxzyb53r5r752q8nh9v9vd150h-udev-rules/69-dm-lvm.rules on line 79, starting at character 32 (',')

The fix[0] for that is fairly trivial.

0: https://git.sr.ht/~graywolf/guix/commit/207794679281c1e7e5f2051719fe5c705506eb23

Have a nice day,
W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#65177: udevd error with lvm-raid array leading to race

[Simon Tournier]

Hi,

On Tue, 31 Oct 2023 at 19:38, Tomas Volf <wolf@wolfsden.cz> wrote:

> I think the patch is wrong.

Is patch v3 from

    https://issues.guix.gnu.org/issue/66586#10

fixing the issue?

Cheers,
simon