BTRFS, LVM, LUKS

BTRFS, LVM, LUKS

[Alexandre Oliva]

Hi,

I'm looking into adopting GuixSD, but I'm kind of a heavy LVM user.  I'm
also slightly concerned about support for BTRFS and LUKS, that I also
rely heavily on.

My first concern is that it's not clear to me what's meant by lack of
LVM support.  I saw patches around April-May 2015, some later
discussions, but not much that would give me much of a clue as to what's
really missing, as in, is it just that the installer doesn't know how to
create logical volumes, that the boot-up code doesn't support root on
lvm, that lvm and device-mapper packages are missing altogether, or that
device-mapper infrastructure is disabled in the kernel?

Full-disk encryption (LUKS) is also a strict requirement for me, and so
is multi-disk BTRFS.

I'm willing and somewhat available to volunteer time and effort to
contribute support for these, but I could use some mentoring in getting
started and on track towards improving what needs to be improved in ways
that are likely to be generally acceptable.

I have not used GuixSD yet, to a large extent due to the stated lack of
LVM support.  I haven't got myself into Guix either *blush*, but maybe I
could get started with it.  I suppose VMs might be a way to get started,
but...  the machines I use are not very powerful, as in, laptops old
enough as to support LibreBoot, so I haven't used virtualization much
myself.

Thanks in advance for any guidance.  I'm lxo on IRC.

-- 
Alexandre Oliva, freedom fighter  he/him   https://FSFLA.org/blogs/lxo
Be the change, be Free!                 FSF Latin America board member
GNU Toolchain Engineer                        Free Software Evangelist
Hay que enGNUrecerse, pero sin perder la terGNUra jamás - Che GNUevara

Re: BTRFS, LVM, LUKS

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 3074 bytes --]

Hello Alexandre,

wellcome to Guix!

Alexandre Oliva <lxoliva@fsfla.org> writes:

[...]

> My first concern is that it's not clear to me what's meant by lack of
> LVM support.  I saw patches around April-May 2015, some later
> discussions, but not much that would give me much of a clue as to what's
> really missing,

Guix is "just" not able to activate/assemble LVM volumes at boot, as a
consequence LVM volumes creation and activation is also missing in the
installer

[...]

> lvm, that lvm and device-mapper packages are missing altogether, or that
> device-mapper infrastructure is disabled in the kernel?

Device mapper is definitely supported
https://www.gnu.org/software/guix/manual/en/html_node/Mapped-Devices.html#Mapped-Devices
it's used for LUKS and mdraid (missing LVM)

> Full-disk encryption (LUKS) is also a strict requirement for me, and so
> is multi-disk BTRFS.

They are working, several of us tested them or are using them in
prodution; I personally manage a physical machine using multi-disk BTRFS
and tested root on BTRFS on LUKS a couple of times on a physical machine

> I'm willing and somewhat available to volunteer time and effort to
> contribute support for these, but I could use some mentoring in getting
> started and on track towards improving what needs to be improved in ways
> that are likely to be generally acceptable.

I cannot mentor, all I can say is that we (I'm also interested in LVM
support for some **legacy** system I manage [1]) must first undestand
how device-mapped device are activated and add support to for LVM ones

In this past discussion Ludovic states:
"It sounds like we’re almost there, I guess."
https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00041.html
but that discussion did not lead to a patch request, nor a wip branch on
guix

As you can see there are no patches for LVM device-mapping
https://debbugs.gnu.org/cgi/pkgreport.cgi?package=guix-patches;archive=both;include=subject%3ALVM

I'm interested in contributing on this but currently I'm just able to
rewiev, test and/or coordinate development :-S

> I have not used GuixSD yet, to a large extent due to the stated lack of
> LVM support.

Please do not be blocked by the lack of LVM support, try start using
Guix on BTRFS on a physical host if you can

> I haven't got myself into Guix either *blush*, but maybe I
> could get started with it.  I suppose VMs might be a way to get started,
> but...  the machines I use are not very powerful, as in, laptops old
> enough as to support LibreBoot, so I haven't used virtualization much
> myself.

All I can say is that to start hacking (that means locally build several
packages or services) on Guix you need enough memory (at least 4GB but
8GB is far better... and use swap!) and enough CPU power (4 cores at
least); if you do not have a powerful host, a VM is suitable just for
testing Guix usage IMHO

[...]

HTH! Gio'.

[1] all new ones _must_ use BTRFS :-)

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: BTRFS, LVM, LUKS

[Alexandre Oliva]

Hello, Giovanni,

On Jun 30, 2019, Giovanni Biscuolo <g@xelera.eu> wrote:

> wellcome to Guix!

Thanks!

> Alexandre Oliva <lxoliva@fsfla.org> writes:

> Guix is "just" not able to activate/assemble LVM volumes at boot

Ok, that doesn't sound too hard to fix.  (famous last words ;-)

> Device mapper is definitely supported

Excellent

> I personally manage a physical machine using multi-disk BTRFS
> and tested root on BTRFS on LUKS a couple of times on a physical machine

Yay!

> "It sounds like we’re almost there, I guess."

Yeah, it looked so nearly finished that I was very surprised it didn't
make it yet.

> Please do not be blocked by the lack of LVM support, try start using
> Guix on BTRFS on a physical host if you can

I might have to make up some room by shrinking LVM partitions ;-)  I
don't think I have any systems that don't have all disks fully allocated
to LVM, other than the yeeloongs.

> All I can say is that to start hacking (that means locally build several
> packages or services) on Guix you need enough memory (at least 4GB but
> 8GB is far better... and use swap!) and enough CPU power (4 cores at
> least)

That won't be a problem, once I get at least some LVM going.  My most
powerful local machines (just as old as the others, but desktops with 6
cores and 24+GB of RAM) should be able to deal with the workload, but
rearranging their disks is even trickier.


[reordered]
> must first undestand how device-mapped device are activated and add
> support to for LVM ones

I'm quite familiar with that myself.  There are basically three ways to
go about it: (i) scan and activate all PV-like devices, (ii) scan and
activate them individually as they come up from udev or whatever, or
(iii) know what you're looking to assemble, and look for the specific
physical volumes

(i) is the simplest, at least as long as you don't have devices that
take a long time ot come up and might cause the scan to time out before
they're there, but I'm told it's not such a good idea for shared-storage
systems (like, multiple hosts connected to the same pool of physical or
virtual storage, with some arbitration among them that can get messed up
if one of them goes about scanning and activating and whatever what
others are already using.

(ii) is probably the sanest, at least after the root fs is fully
activated, since removable storage might be plugged in at any time, and
the infrastructure to support scanning and activating it automatically
then is pretty much the same that activates the initially-available
volumes, though the latter might take some simulated udev events.

(iii) sounds a little more in line with what I understand GuixSD system
configuration is about, but...  though it's nice to have a config file
describing how the storage is set up, that kind of obviates the
flexibility of lvm and could make things more difficult for storage
disaster recovery.


Anyway, I guess it would make most sense to at least start building up
on existing practice.  How does Guix currently bring up multi-device
root filesystems (btrfs, mdraid, ...), and any recursive combinations of
mdraid, dmcrypt, etc?  I suppose it would make sense to stick to similar
logic in bringing up physical volumes towards assembling a volume group
containing the root logical volume, bearing in mind that any of these
might be also be mdraid, dmraid, dmcrypt...  So a single lvm vgscan,
while covering the simplest configurations, would not quite go all the
way to full generality, which, in the GNU spirit of avoiding arbitrary
limitations, if not quite in in the strict sense of the letter, is what
I assume GuixSD would aim for.  Is that so?

-- 
Alexandre Oliva, freedom fighter  he/him   https://FSFLA.org/blogs/lxo
Be the change, be Free!                 FSF Latin America board member
GNU Toolchain Engineer                        Free Software Evangelist
Hay que enGNUrecerse, pero sin perder la terGNUra jamás - Che GNUevara

Re: BTRFS, LVM, LUKS

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

Alexandre Oliva <lxoliva@fsfla.org> writes:

[...]

> Anyway, I guess it would make most sense to at least start building up
> on existing practice.  How does Guix currently bring up multi-device
> root filesystems (btrfs, mdraid, ...),

With BTRFS multi device activation is built in the Linux kernel AFAIU,
so no need to "activate" it, just mount one of the devices of a
multi-device BTRFS filesystem (using uuid id more resilient)

I cannot help much with mdraid since I still had not the chance to
study how device-mapper devices are activated; AFAIU the code is in (gnu
system mapped-devices)

> and any recursive combinations of mdraid, dmcrypt, etc?

Recursive? Do you mean LUKS on mdraid on LUKS? Or mdraid on LUKS on
mdraid? AFAIU this kind of *nesting* should be supported but maybe just
a complication :-)

raid-device-mapping and luks-device-mapping are a type of
mapped-devices currently supported, you can combine them as you like to
assemble a device to be used for your filesystems

Filesystems can have dependencies on other filesystems or mapped-devices
(see "dependencies" member of file-system)

I hope this clarifies how Guix assemble its filesystems.

[...]

Happy Guix! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: BTRFS, LVM, LUKS

[Alexandre Oliva]

On Jul  5, 2019, Giovanni Biscuolo <g@xelera.eu> wrote:

> With BTRFS multi device activation is built in the Linux kernel AFAIU,

I'm pretty sure btrfs dev scan or similar must introduce each of the
filesystem components to the kernel before the kernel will use them.

> I cannot help much with mdraid since I still had not the chance to
> study how device-mapper devices are activated

There used to be in-kernel scanning in mdraid, but I recall the plan as
of 15+ years ago was to phase that out and take care of it all in
userland.

>> and any recursive combinations of mdraid, dmcrypt, etc?

> Recursive? Do you mean LUKS on mdraid on LUKS?

Well, LUKS doesn't make much sense to duplicate indeed, though there's
no reason to prohibit it (say, one could be mirroring a fully-encrypted
disk onto a logical volume that's fully encrypted itself), but building
volume groups or raid components out of logical volumes is occasionally
useful, even when the logical volumes are backed by other raid devices.

> Filesystems can have dependencies on other filesystems or mapped-devices
> (see "dependencies" member of file-system)

> I hope this clarifies how Guix assemble its filesystems.

To some extent.  I meant to ask about the dynamics during boot-up rather
than the static description in the configuration, but that's very useful
information to get me going in a not too distant future.

(I'll be too busy with other stuff, like finishing a new lecture to be
delivered by the end of the month, to jump into this right away; I
wanted to, in order to use guix to deliver the speech, but I doubt I'm
going to be able to make the jump in time ;-(

-- 
Alexandre Oliva, freedom fighter  he/him   https://FSFLA.org/blogs/lxo
Be the change, be Free!                 FSF Latin America board member
GNU Toolchain Engineer                        Free Software Evangelist
Hay que enGNUrecerse, pero sin perder la terGNUra jamás - Che GNUevara

Re: BTRFS, LVM, LUKS

[Ludovic Courtès]

Hi,

Alexandre Oliva <lxoliva@fsfla.org> skribis:

> On Jun 30, 2019, Giovanni Biscuolo <g@xelera.eu> wrote:

[...]

>> "It sounds like we’re almost there, I guess."

Ahem, we have a use case for the right to be forgotten.  :-)

> Yeah, it looked so nearly finished that I was very surprised it didn't
> make it yet.

I think the mapped device API was very new back then, and perhaps we
weren’t initially quite sure where we were going.

Regardless, I’m all in favor of reviving this patch set, it should be
easier this time around!

There are also people like Giovanni who have pretty fancy setups with
Btrfs, if that’s an option for you.

Welcome!

Ludo’.

Re: BTRFS, LVM, LUKS

[Alexandre Oliva]

On Jul  5, 2019, Ludovic Courtès <ludo@gnu.org> wrote:

> Regardless, I’m all in favor of reviving this patch set, it should be
> easier this time around!

Thanks.  Should I contact the original author for copyright purposes, or
may I assume it was contributed, even if not yet integrated, and take it
from there?

> There are also people like Giovanni who have pretty fancy setups with
> Btrfs, if that’s an option for you.

Not exactly an option, but a complement.  I actually rely on both, with
different purposes.

> Welcome!

Thanks ;-)  I'm very excited to be seriously looking into taking this
step ;-)

-- 
Alexandre Oliva, freedom fighter  he/him   https://FSFLA.org/blogs/lxo
Be the change, be Free!                 FSF Latin America board member
GNU Toolchain Engineer                        Free Software Evangelist
Hay que enGNUrecerse, pero sin perder la terGNUra jamás - Che GNUevara

Re: BTRFS, LVM, LUKS

[Ludovic Courtès]

Hello,

Alexandre Oliva <lxoliva@fsfla.org> skribis:

> On Jul  5, 2019, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Regardless, I’m all in favor of reviving this patch set, it should be
>> easier this time around!
>
> Thanks.  Should I contact the original author for copyright purposes, or
> may I assume it was contributed, even if not yet integrated, and take it
> from there?

The latter, IMO.

Thanks for taking a look!

Ludo’.

Re: BTRFS, LVM, LUKS

[David Larsson]

Hi lxo,
I can give some advice reg. libreboot+luks+btrfs, but not reg. LVM and I haven't tested a a btrfs multi-disk setup with GuixSD either unfortunately. 

Using btrfs and luks on a librebooted laptop works well.

> Full-disk encryption (LUKS) is also a strict requirement for me, and so
> is multi-disk BTRFS.

When using libreboot and GuixSD, I suggest that you have in the libreboot grub:

cryptomount -a
configfile /boot/grub.cfg 

in order to get all the Guix rollback features that are related to the grub-menu entries without needing to reflash after each guix system reconfigure.

But I suggest that /boot/grub.cfg is a symlink to the /boot in the Guix root subvolume. So the full path to Guix's boot.cfg would be:

/guix_rw/boot/grub.cfg
 
By loading the Guix grub from libreboot grub like that you can reboot to a new btrfs snapshot such as /guix_rw_test before doing risky things just by changing the destination file of the symlink.

The one issue I have had with this is that the GuixSD system reconfigure command generates grub-entries with the current relative path to initrd etc. so I had to write a small bash-script that replaced the grub-entries with the full paths, like this:

function grubfix(){
        sudo sed -i "s/linux\ \/gnu/linux\ \/guix_rw\/gnu/g" "$1"
        sudo sed -i "s/initrd\ \/gnu/initrd\ \/guix_rw\/gnu/g" "$1"
        sudo sed -i "s/--set\ \/gnu/--set\ \/guix_rw\/gnu/g" "$1"
}

which I then wrapped to a bash function called guix-update() that invokes guix system reconfigure and ends with the grubfix() above.

> 
> I have not used GuixSD yet, to a large extent due to the stated lack of
> LVM support. I haven't got myself into Guix either *blush*, but maybe I
> could get started with it. I suppose VMs might be a way to get started,
> but... the machines I use are not very powerful, as in, laptops old
> enough as to support LibreBoot, so I haven't used virtualization much
> myself.

I ran GuixSD on an old x200 for a few a years and had a really painful time with overheating issues. This happens as soon as you have to build anything from source which happens every now and then. I would primarily recommend that you get a more powerful laptop like one of the Librems.

There are though a few ways to mitigate small laptop heating issues:
  - run a Guix substitute server yourself on a more powerful machine (quite easy setup).
  - make sure to use several of the freely available substitute servers (e.g. berlin.guixsd.org)
  - underclocking
  - limiting the reconfigure to fewer cores

To limit core cores and add additional substitute servers, you can add to your system services list something like below:

 (modify-services
                (guix-service-type config =>
                        (guix-configuration
                          (inherit config)
                          (substitute-urls
                            (cons *
 "https://mirror.hydra.gnu.org" "https://mirror.guixsd.org"
		              "https://berlin.guixsd.org"
		              %default-substitute-urls))))
;                           (extra-options                                                                                                                     
;              	              '("--cores=1"))))) ; to avoid overheating	from build-processes                                                                   
             %base-services)))

> 
> Thanks in advance for any guidance. I'm lxo on IRC.
> 
> -- 
> Alexandre Oliva, freedom fighter he/him FSFLA.org/blogs/lxo
> Be the change, be Free! FSF Latin America board member
> GNU Toolchain Engineer Free Software Evangelist
> Hay que enGNUrecerse, pero sin perder la terGNUra jamás - Che GNUevara

I wish you the best of luck with GuixSD!

// David

Re: BTRFS, LVM, LUKS

[Alexandre Oliva]

On Jun 30, 2019, David Larsson <mail@selfhosted.xyz> wrote:

> When using libreboot and GuixSD, I suggest that you have in the libreboot grub:

> cryptomount -a
> configfile /boot/grub.cfg 

Thanks, yeah, that's pretty much how I deal with the distro I currently
use.

> But I suggest that /boot/grub.cfg is a symlink to the /boot in the
> Guix root subvolume.

*nod*, same here.  It has given me some headaches with grub probing and
config rewriting, which from what you write I conclude I may have to
figure out again ;-)

>   - run a Guix substitute server yourself on a more powerful machine (quite easy setup).

The term "substitute server" doesn't sound familiar to me.  From what
you suggest, I guess that's a Guix concept related with offloading
system rebuilds to it, is that so?

> I wish you the best of luck with GuixSD!

Thanks!

-- 
Alexandre Oliva, freedom fighter  he/him   https://FSFLA.org/blogs/lxo
Be the change, be Free!                 FSF Latin America board member
GNU Toolchain Engineer                        Free Software Evangelist
Hay que enGNUrecerse, pero sin perder la terGNUra jamás - Che GNUevara

Re: BTRFS, LVM, LUKS

[Christopher Lemmer Webber]

Alexandre Oliva writes:

> Full-disk encryption (LUKS) is also a strict requirement for me, and so
> is multi-disk BTRFS.

Note that full disk encryption does work without LVM in Guix, though you
do need to then pretty much put everything on one partition :)

I think you can also do full disk encryption from libreboot by
decrypting the whole disk from GRUB.  I forget how that works, though.

Welcome to Guix, btw!

Re: BTRFS, LVM, LUKS

[Ludovic Courtès]

Hi,

Christopher Lemmer Webber <cwebber@dustycloud.org> skribis:

> Alexandre Oliva writes:
>
>> Full-disk encryption (LUKS) is also a strict requirement for me, and so
>> is multi-disk BTRFS.
>
> Note that full disk encryption does work without LVM in Guix, though you
> do need to then pretty much put everything on one partition :)

Yeah that’s what I do.  For the record, there’s an example of LUKS
device mapping here:

  https://gnu.org/s/guix/manual/en/html_node/Using-the-Configuration-System.html#System-Services

and the ‘luks-device-mapping’ doc is at:

  https://gnu.org/s/guix/manual/en/html_node/Mapped-Devices.html#index-LUKS

Ludo’.