guix system init with mounted btrfs

guix system init with mounted btrfs

[Matt Huszagh]

I'm encountering the error "rmdir: Device or resource busy" when I
attempt to perform guix system init on a mount point that has btrfs
subvolumes. Here's the process I've used for creating and mounting the
subvolumes prior to running `guix system init`:

mount -t btrfs -o compress=lzo LABEL=btrfs /mnt
btrfs subvolume create /mnt/@guixsd
btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@snapshots
btrfs subvolume create /mnt/@grub
btrfs subvolume create /mnt/@guix
btrfs subvolume create /mnt/@gnu
umount /mnt
o_btrfs=defaults,X-mount.mkdir,compress=lzo,ssd,noatime

mount -o subvol=@guixsd,$o_btrfs LABEL=btrfs /mnt
mount -o subvol=@home,$o_btrfs LABEL=btrfs /mnt/home
mount -o subvol=@snapshots,$o_btrfs LABEL=btrfs /mnt/.snapshots
mount -o subvol=@grub,$o_btrfs LABEL=btrfs /mnt/boot/grub
mount -o subvol=@guix,$o_btrfs LABEL=btrfs /mnt/var/guix
mount -o subvol=@gnu,$o_btrfs LABEL=btrfs /mnt/gnu
mount -o X-mount.mkdir /dev/nvme0n1p1 /mnt/boot/efi

What have I done wrong here?

thx

Re: guix system init with mounted btrfs

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

Matt,

[If you sent two mails because your first didn't immediately 
arrive: it was probably held for moderation since you appear to be 
new here.  Welcome!]

Matt Huszagh wrote:
> I'm encountering the error "rmdir: Device or resource busy" when 
> I
> attempt to perform guix system init on a mount point that has 
> btrfs
> subvolumes.

So something's (preventatively?) deleting a directory that has 
something mounted over or in it, or is a subdirectory itself 
(although that seems unlikely in this layout).

I had a dig around, didn't find any likely culprits, but this 
could be happening anywhere.  Does ‘guix system init’ print 
anything else?

Don't waste *too* much time debugging your error message, though. 
Guix System doesn't have any code to handle fancy subvolume 
swappery like this.  I'm afraid you'll have to drop the @-scheme 
entirely.

> mount -o subvol=@guixsd,$o_btrfs LABEL=btrfs /mnt
> mount -o subvol=@gnu,$o_btrfs LABEL=btrfs /mnt/gnu

The two above are especially problematic: Guix's GRUB looks only 
for /gnu/store/…-linux/, which is now two indirections away.

Guix System *probably* supports⁰ ‘in-place’ subvolumes like

  btrfs subvol create /gnu/store

only because it doesn't need to support them: they're just 
directories.

> mount -o subvol=@grub,$o_btrfs LABEL=btrfs /mnt/boot/grub

Does GRUB support magically finding itself elsewhere like this? 
Does this work on other distributions?  If so, do you know how?

…

You're damn right that this needs to be improved.  Hands welcome 
:-)

Kind regards,

T G-R

[0]: At least I think I did so once but never made use of its 
subvolliness, so I stopped doing so.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: guix system init with mounted btrfs

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 298 bytes --]

Erk,

Tobias Geerinckx-Rice wrote:
> So something's (preventatively?) deleting a directory that has
> something mounted over or in it, or is a subdirectory itself 
> (although
> that seems unlikely in this layout).     ^^^^^^^^^^^^

‘subvolume’, of course…

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: guix system init with mounted btrfs

[Matt Huszagh]

Thanks for the reply/suggestions Tobias!

Tobias Geerinckx-Rice <me@tobias.gr> writes:
>> mount -o subvol=@guixsd,$o_btrfs LABEL=btrfs /mnt
>> mount -o subvol=@gnu,$o_btrfs LABEL=btrfs /mnt/gnu
>
> The two above are especially problematic: Guix's GRUB looks only for /gnu/store/…-linux/, which is
> now two indirections away.
>
> Guix System *probably* supports⁰ ‘in-place’ subvolumes like
>
>  btrfs subvol create /gnu/store
>
> only because it doesn't need to support them: they're just directories.
>
>> mount -o subvol=@grub,$o_btrfs LABEL=btrfs /mnt/boot/grub
>
> Does GRUB support magically finding itself elsewhere like this? Does this work on other
> distributions?  If so, do you know how?

Ok, so I've simplified the subvolume layout somewhat:
mount -t btrfs -o compress=lzo LABEL=btrfs /mnt
btrfs subvolume create /mnt/@guixsd
btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@snapshots
umount /mnt
o_btrfs=defaults,X-mount.mkdir,compress=lzo,ssd,noatime

mount -o subvol=@guixsd,$o_btrfs LABEL=btrfs /mnt
mount -o subvol=@home,$o_btrfs LABEL=btrfs /mnt/home
mount -o subvol=@snapshots,$o_btrfs LABEL=btrfs /mnt/.snapshots
mount -o X-mount.mkdir /dev/nvme0n1p1 /mnt/boot/efi

guix system init succeeds with this layout! So it seems like you're
right and guix only currently supports in-place subvolumes.

Additionally, this appears to fix another probable error which I didn't
originally mention. grub.cfg was omitting path components. So the linux
command was something like linux /store/... instead of
linux /gnu/store/...

That's now fixed.

I'm now encountering another error when attempting to boot however. I
have 2 luks-encrypted SSDs and grub only prompts me for one
passphrase. Since I'm using a BTRFS filesystem spanning partitions from
both devices, I'm sent to a grub rescue prompt after decrypting the
first device. I took a look through the GRUB reference manual and it
looks like decryption is performed by the "cryptomount"
command. However, I don't see any reference to "cryptomount" or
"mkconfig" (for grub-mkconfig) in the guix source code. Additionally,
when I look at the generated grub.cfg file I don't see cryptomount
anywhere. Still, it does at least prompt me for one passphrase.

I've listed both devices as mapped-devices:

(define cryptnvme
  (mapped-device
   (source (uuid "ee3b58e3-e3a1-4d83-8306-060fcd102761"))
   (target "cryptnvme")
   (type luks-device-mapping)))

(define cryptsd
  (mapped-device
   (source (uuid "4b61ed37-f628-4d78-b6eb-099be4a25b15"))
   (target "cryptsd")
   (type luks-device-mapping)))

...

 (mapped-devices (list cryptnvme cryptsd))

Any thoughts on what might be causing that?

> You're damn right that this needs to be improved.  Hands welcome :-)

Once I get the simple subvolume layout working I'll take a look at
getting the more complex setup to work and report back!

Thanks
Matt

Re: guix system init with mounted btrfs

[Matt Huszagh]

Tobias Geerinckx-Rice <me@tobias.gr> writes:

Matt Huszagh <huszaghmatt@gmail.com> writes:
> Ok, so I've simplified the subvolume layout somewhat:
> mount -t btrfs -o compress=lzo LABEL=btrfs /mnt
> btrfs subvolume create /mnt/@guixsd
> btrfs subvolume create /mnt/@home
> btrfs subvolume create /mnt/@snapshots
> umount /mnt
> o_btrfs=defaults,X-mount.mkdir,compress=lzo,ssd,noatime
>
> mount -o subvol=@guixsd,$o_btrfs LABEL=btrfs /mnt
> mount -o subvol=@home,$o_btrfs LABEL=btrfs /mnt/home
> mount -o subvol=@snapshots,$o_btrfs LABEL=btrfs /mnt/.snapshots
> mount -o X-mount.mkdir /dev/nvme0n1p1 /mnt/boot/efi

I think I was a bit hasty in saying this worked. Another possible reason
is guix system init doesn't try to do anything with these directories
anyway so maybe the fact that these subvolumes are "supported" is a
false positive.

I've tried an even simpler layout: no subvolumes, just

mount -t btrfs -o defaults,compress=lzo,ssd,noatime LABEL=btrfs /mnt
mount -o X-mount.mkdir /dev/nvme0n1p1 /mnt/boot/efi

However, I'm experiencing the same decryption error with this setup.

Re: guix system init with mounted btrfs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1951 bytes --]

On Sun, Jun 23, 2019 at 02:10:48PM -0700, Matt Huszagh wrote:
> Tobias Geerinckx-Rice <me@tobias.gr> writes:
> 
> Matt Huszagh <huszaghmatt@gmail.com> writes:
> > Ok, so I've simplified the subvolume layout somewhat:
> > mount -t btrfs -o compress=lzo LABEL=btrfs /mnt
> > btrfs subvolume create /mnt/@guixsd
> > btrfs subvolume create /mnt/@home
> > btrfs subvolume create /mnt/@snapshots
> > umount /mnt
> > o_btrfs=defaults,X-mount.mkdir,compress=lzo,ssd,noatime
> >
> > mount -o subvol=@guixsd,$o_btrfs LABEL=btrfs /mnt
> > mount -o subvol=@home,$o_btrfs LABEL=btrfs /mnt/home
> > mount -o subvol=@snapshots,$o_btrfs LABEL=btrfs /mnt/.snapshots
> > mount -o X-mount.mkdir /dev/nvme0n1p1 /mnt/boot/efi
> 
> I think I was a bit hasty in saying this worked. Another possible reason
> is guix system init doesn't try to do anything with these directories
> anyway so maybe the fact that these subvolumes are "supported" is a
> false positive.
> 
> I've tried an even simpler layout: no subvolumes, just
> 
> mount -t btrfs -o defaults,compress=lzo,ssd,noatime LABEL=btrfs /mnt
> mount -o X-mount.mkdir /dev/nvme0n1p1 /mnt/boot/efi
> 
> However, I'm experiencing the same decryption error with this setup.
> 

I spent some time yesterday with the arch wiki and I have an idea.
assuming you're using %desktop-services, inside services:

(modify-services %desktop-services
  (udev-service-type config =>
                     (udev-configuration
                     (inherit config)
                     (rules (list lvm2 fuse alsa-utils crda btrfs)))))

Perhaps adding the udev rules for btrfs will force it to run 'btrfs
device scan' or whatever the magic is that it apparently isn't running
now.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: guix system init with mounted btrfs

[Matt Huszagh]

Efraim Flashner <efraim@flashner.co.il> writes:
> I spent some time yesterday with the arch wiki and I have an idea.
> assuming you're using %desktop-services, inside services:
>
> (modify-services %desktop-services
>   (udev-service-type config =>
>                      (udev-configuration
>                      (inherit config)
>                      (rules (list lvm2 fuse alsa-utils crda btrfs)))))
>
> Perhaps adding the udev rules for btrfs will force it to run 'btrfs
> device scan' or whatever the magic is that it apparently isn't running
> now.

Thanks for the suggestion Efraim. I don't currently use desktop
services, just base-services at the moment. However, I'm concerned there
may still be an issue with this modification. I'm currently only getting
prompted for 1 of 2 LUKS device passphrases, which I believe is an issue
with the grub bootloader (which in my case is an efi shell program). I'm
getting stuck at the grub rescue prompt and so I don't even get to the
initial ramdisk stage. Is that where these services kick in, or does
that happen later when the full kernel is loaded? When I mount the btrfs
root fs manually, I have to decrypt both devices before I can run btrfs
device scan.

To address the grub issue I've tried doing:
grub rescue> insmod luks
grub rescue> cryptomount -a

However, that doesn't appear to have any effect. I've also tried
modifying grub.cfg directly and reinstalling the bootloader by chrooting
and running grub-install, but that doesn't work for reasons that seem to
be related to the guix way of doing things. Since I'm still new to Guix,
I don't know how one might make custom modifications to the cfg file.

Re: guix system init with mounted btrfs

[Matt Huszagh]

Efraim Flashner <efraim@flashner.co.il> writes:
> I spent some time yesterday with the arch wiki and I have an idea.
> assuming you're using %desktop-services, inside services:
>
> (modify-services %desktop-services
>   (udev-service-type config =>
>                      (udev-configuration
>                      (inherit config)
>                      (rules (list lvm2 fuse alsa-utils crda btrfs)))))
>
> Perhaps adding the udev rules for btrfs will force it to run 'btrfs
> device scan' or whatever the magic is that it apparently isn't running
> now.

Efraim, is btrfs supposed to btrfs-progs here? I believe rules takes
package names right? In any event I tried this and it didn't work for me.
Did you have success with it?

On Mon, Jun 24, 2019 at 7:51 AM Matt Huszagh <huszaghmatt@gmail.com> wrote:

> Efraim Flashner <efraim@flashner.co.il> writes:
> > I spent some time yesterday with the arch wiki and I have an idea.
> > assuming you're using %desktop-services, inside services:
> >
> > (modify-services %desktop-services
> >   (udev-service-type config =>
> >                      (udev-configuration
> >                      (inherit config)
> >                      (rules (list lvm2 fuse alsa-utils crda btrfs)))))
> >
> > Perhaps adding the udev rules for btrfs will force it to run 'btrfs
> > device scan' or whatever the magic is that it apparently isn't running
> > now.
>
> Thanks for the suggestion Efraim. I don't currently use desktop
> services, just base-services at the moment. However, I'm concerned there
> may still be an issue with this modification. I'm currently only getting
> prompted for 1 of 2 LUKS device passphrases, which I believe is an issue
> with the grub bootloader (which in my case is an efi shell program). I'm
> getting stuck at the grub rescue prompt and so I don't even get to the
> initial ramdisk stage. Is that where these services kick in, or does
> that happen later when the full kernel is loaded? When I mount the btrfs
> root fs manually, I have to decrypt both devices before I can run btrfs
> device scan.
>
> To address the grub issue I've tried doing:
> grub rescue> insmod luks
> grub rescue> cryptomount -a
>
> However, that doesn't appear to have any effect. I've also tried
> modifying grub.cfg directly and reinstalling the bootloader by chrooting
> and running grub-install, but that doesn't work for reasons that seem to
> be related to the guix way of doing things. Since I'm still new to Guix,
> I don't know how one might make custom modifications to the cfg file.
>


-- 

*Matt Huszagh*
Emerging Markets Trader | JP Morgan

o: (212) 834-4869
c: (847) 313-5082
LinkedIn <http://www.linkedin.com/in/matt-huszagh-987b3a37/>

Re: guix system init with mounted btrfs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3566 bytes --]

On Mon, Jun 24, 2019 at 10:54:37AM -0700, Matt Huszagh wrote:
> Efraim Flashner <efraim@flashner.co.il> writes:
> > I spent some time yesterday with the arch wiki and I have an idea.
> > assuming you're using %desktop-services, inside services:
> >
> > (modify-services %desktop-services
> >   (udev-service-type config =>
> >                      (udev-configuration
> >                      (inherit config)
> >                      (rules (list lvm2 fuse alsa-utils crda btrfs)))))
> >
> > Perhaps adding the udev rules for btrfs will force it to run 'btrfs
> > device scan' or whatever the magic is that it apparently isn't running
> > now.
> 
> Efraim, is btrfs supposed to btrfs-progs here? I believe rules takes
> package names right? In any event I tried this and it didn't work for me.
> Did you have success with it?
> 

I haven't tried it yet, I'm somehow running low on physical machines and
I haven't set up a VM to test it yet. The two places I remember seeing
something about it was with Gentoo¹ and with Arch², which is where I got
the udev suggestion for btrfs, from the 64-btrfs.rule file.

¹ https://forums.gentoo.org/viewtopic-t-1010878.html
² https://bbs.archlinux.org/viewtopic.php?id=189845

> On Mon, Jun 24, 2019 at 7:51 AM Matt Huszagh <huszaghmatt@gmail.com> wrote:
> 
> > Efraim Flashner <efraim@flashner.co.il> writes:
> > > I spent some time yesterday with the arch wiki and I have an idea.
> > > assuming you're using %desktop-services, inside services:
> > >
> > > (modify-services %desktop-services
> > >   (udev-service-type config =>
> > >                      (udev-configuration
> > >                      (inherit config)
> > >                      (rules (list lvm2 fuse alsa-utils crda btrfs)))))
> > >
> > > Perhaps adding the udev rules for btrfs will force it to run 'btrfs
> > > device scan' or whatever the magic is that it apparently isn't running
> > > now.
> >
> > Thanks for the suggestion Efraim. I don't currently use desktop
> > services, just base-services at the moment. However, I'm concerned there
> > may still be an issue with this modification. I'm currently only getting
> > prompted for 1 of 2 LUKS device passphrases, which I believe is an issue
> > with the grub bootloader (which in my case is an efi shell program). I'm
> > getting stuck at the grub rescue prompt and so I don't even get to the
> > initial ramdisk stage. Is that where these services kick in, or does
> > that happen later when the full kernel is loaded? When I mount the btrfs
> > root fs manually, I have to decrypt both devices before I can run btrfs
> > device scan.
> >
> > To address the grub issue I've tried doing:
> > grub rescue> insmod luks
> > grub rescue> cryptomount -a
> >
> > However, that doesn't appear to have any effect. I've also tried
> > modifying grub.cfg directly and reinstalling the bootloader by chrooting
> > and running grub-install, but that doesn't work for reasons that seem to
> > be related to the guix way of doing things. Since I'm still new to Guix,
> > I don't know how one might make custom modifications to the cfg file.
> >
> 
> 
> -- 
> 
> *Matt Huszagh*
> Emerging Markets Trader | JP Morgan
> 
> o: (212) 834-4869
> c: (847) 313-5082
> LinkedIn <http://www.linkedin.com/in/matt-huszagh-987b3a37/>

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]