Use genimage for disk-image creation.

Use genimage for disk-image creation.

[Mathieu Othacehe]

Hello,

I recently spent some time generating a lot of Guix System disk-images
(embedded system images, installation tests). The mechanism Guix and Nix
are using: spawning a VM to create the disk-image does not scale at all
when the disk-image size increase.

So, I looked at what the competition does. Buildroot is using
genimage[1] to generate disk-images. Yocto is using wic[2], that's based
on mic[3]. Both are able to create disk-images as unprivileged users,
without resorting to VM.

While, Yocto solution is tied to the rest of their framework, genimage
is standalone. In fact, Danny has already packaged it and made some
research about it.

I think we could use genimage to replace our current mechanism to
generate disk-images. Here's a small proof of concept.

Let say I want to create a disk-image with one ext4 partition starting
at offset 10M. I can write the following genimage config file:

--8<---------------cut here---------------start------------->8---
image system {
	hdimage {}

	partition rootfs {
		partition-type = 0x83
		image = "rootfs.ext4"
		size = 8G
		offset = 10M
	}
}

image rootfs.ext4 {
        name = "rootfs"
        ext4 {
                label = "rootfs"
		use-mke2fs = true
        }
        size = 8G
        mountpoint = "/"
}
--8<---------------cut here---------------end--------------->8---

and run the command:

--8<---------------cut here---------------start------------->8---
fakeroot genimage --config ~/tmp/genimage.cfg --rootpath=/home/mathieu/image-root/
--8<---------------cut here---------------end--------------->8---

where image-root is a directory containing the result of a `guix system
init`. The directory size is about 6GiB.

It takes 8 minutes to generate this disk-image, versus 2h30 using `guix
system disk-image`.

I'm aware that this might not be a fair comparison but, I think its
already significant.

Danny, Ludo, WDYT? Could we modify "system-disk-image" to use genimage
as a backend instead of spawning a VM?

Thanks,

Mathieu

[1]: https://github.com/pengutronix/genimage
[2]: https://www.yoctoproject.org/docs/2.4.2/dev-manual/dev-manual.html#creating-partitioned-images-using-wic
[3]: https://source.tizen.org/documentation/reference/mic-image-creator
[4]: https://lists.gnu.org/archive/html/guix-devel/2019-09/msg00009.html

Re: Use genimage for disk-image creation.

[Vincent Legoll]

FYI, I'm investigating the test suite failures for the newer
releases of genimage (v11 (+1 FAIL) & v12 (+2 FAIL))...

I think I'll report / PR them upstream before updating
guix's version, to avoid unnecessary churn.

-- 
Vincent Legoll

Re: Use genimage for disk-image creation.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

On Thu, Mar 26, 2020 at 10:55:04AM +0100, Vincent Legoll wrote:
> FYI, I'm investigating the test suite failures for the newer
> releases of genimage (v11 (+1 FAIL) & v12 (+2 FAIL))...
> 
> I think I'll report / PR them upstream before updating
> guix's version, to avoid unnecessary churn.
> 

I found that updating Guix's version of mtools also spawned errors on
genimage on aarch64/armhf (IIRC), so that might also be something worth
looking into.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Use genimage for disk-image creation.

[Ludovic Courtès]

Hi Mathieu!

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> Let say I want to create a disk-image with one ext4 partition starting
> at offset 10M. I can write the following genimage config file:
>
> image system {
> 	hdimage {}
>
> 	partition rootfs {
> 		partition-type = 0x83
> 		image = "rootfs.ext4"
> 		size = 8G
> 		offset = 10M
> 	}
> }
>
> image rootfs.ext4 {
>         name = "rootfs"
>         ext4 {
>                 label = "rootfs"
> 		use-mke2fs = true
>         }
>         size = 8G
>         mountpoint = "/"
> }
>
>
> and run the command:
>
> fakeroot genimage --config ~/tmp/genimage.cfg --rootpath=/home/mathieu/image-root/
>
> where image-root is a directory containing the result of a `guix system
> init`. The directory size is about 6GiB.
>
> It takes 8 minutes to generate this disk-image, versus 2h30 using `guix
> system disk-image`.
>
> I'm aware that this might not be a fair comparison but, I think its
> already significant.
>
> Danny, Ludo, WDYT? Could we modify "system-disk-image" to use genimage
> as a backend instead of spawning a VM?

I’m completely sold to the idea.  :-)

Apparently ‘genimage’ supports many file systems, including ext[234] and
ISO9660, which are the two formats we support via ‘--file-system-type’.
It does not support Btrfs, but ‘guix system disk-image’ doesn’t support
it either so far.

Looking at ‘image-ext2.c’ reveals that genimage actually just shells out
to mke2fs.  Indeed, I discovered that ‘mke2fs -d /my/root’ copies
/my/root as the image’s root directory.  Likewise, for ISO, it just
shells out to ‘genisoimage’.

So I think that we could avoid ‘genimage’ altogether and implement
similar functionality for ext4/ISO in (gnu build disk-image).

WDYT?

Thanks,
Ludo’.

Re: Use genimage for disk-image creation.

[Vincent Legoll]

> I’m completely sold to the idea.  :-)

Yep, LGTM too

> Apparently ‘genimage’ supports many file systems, including ext[234] and
> ISO9660, which are the two formats we support via ‘--file-system-type’.
> It does not support Btrfs, but ‘guix system disk-image’ doesn’t support
> it either so far.
>
> Likewise, for ISO, it just shells out to ‘genisoimage’.

You probably meant "mkisoimage" ?

> So I think that we could avoid ‘genimage’ altogether and implement
> similar functionality for ext4/ISO in (gnu build disk-image).

Do you anticipate less or similar work in reimplementing rather than
reusing ? Genuinely asking, because we may want to support other
image formats in the future, so reusing may have its interest too...

Or is there another reason that I'm not seing, like maybe avoiding some
overhead (probably not the fork/execs going through genisoimage though)...

-- 
Vincent Legoll

Re: Use genimage for disk-image creation.

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

Hi Mathieu,

from the standpoint of ARM it would be really good to be able to reuse
genimage config files from buildroot.

In fact, partition layout is a major pain in the ass to get right on ARM.
If we want to support a large number of ARM platforms, that would
mean the partitioning would be fixed and not user-modifyable anyway.

> Danny, Ludo, WDYT? Could we modify "system-disk-image" to use genimage
> as a backend instead of spawning a VM?

It depends on whether genimage supports all filesystems we care about.
I suspect that it does, though.

Genimage images tend to be reproducible btw, so it's kinda nice.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Use genimage for disk-image creation.

[Mathieu Othacehe]

Hello Vincent,

> FYI, I'm investigating the test suite failures for the newer
> releases of genimage (v11 (+1 FAIL) & v12 (+2 FAIL))...
>
> I think I'll report / PR them upstream before updating
> guix's version, to avoid unnecessary churn.

Ok good, let us know how it goes :) I should have mentioned that I had
to disable the tests to be able to build genimage.

Thanks,

Mathieu

Re: Use genimage for disk-image creation.

[Vincent Legoll]

Hello

> > FYI, I'm investigating the test suite failures for the newer
> > releases of genimage (v11 (+1 FAIL) & v12 (+2 FAIL))...
> >
> > I think I'll report / PR them upstream before updating
> > guix's version, to avoid unnecessary churn.
>
> Ok good, let us know how it goes :) I should have mentioned that I had
> to disable the tests to be able to build genimage.

I've sent an update to v11, this afternoon, since the fix for that one
was simple enough. But I've yet to receive the ack from debbugs
for it. That's strange, I usually get that ack in just a few minutes...

For v12, I'll need to dig further, even if the test suite failure is probably
harmless...

Details in the guix-patches email for v11, when that surfaces.

--
Vincent Legoll

Re: Use genimage for disk-image creation.

[Ludovic Courtès]

Hi,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> from the standpoint of ARM it would be really good to be able to reuse
> genimage config files from buildroot.

Well, we can have a parser.  :-)

From an engineering viewpoint, it seems weird to add a C parser that
basically shells out to other tools when we could just invoke those
commands directly and hopefully get more control and better error
reporting.

> In fact, partition layout is a major pain in the ass to get right on ARM.
> If we want to support a large number of ARM platforms, that would
> mean the partitioning would be fixed and not user-modifyable anyway.

Oh, really?  I’m surprised partitioning causes problems (though I’m
not familiar with embedded dev!).

Ludo’.

Re: Use genimage for disk-image creation.

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

Hi Ludo,

On Sun, 29 Mar 2020 16:44:39 +0200
Ludovic Courtès <ludo@gnu.org> wrote:

> Oh, really?  I’m surprised partitioning causes problems (though I’m
> not familiar with embedded dev!).

Well, maybe not that bad, but it's pretty bad.

It's because the boot sector for ARM, for some unfathomable reason, is not
the first sector but has basically a vendor-dependent sector number somewhere
in the middle of the disk :P

(with the "boot sector" I mean the sector(s) where usually u-boot resides)

So there are all kinds of stupid problems like sometimes you have to reduce
the size of the GPT partition table (TABLE!) because it would write right over
the actual boot sector of the platform otherwise.  For that vendor.  Other
vendors do other silly things.

Parted doesn't really have support for that stuff either, and I while I
brought it up with them[1], I can't think of a practical way to fix it
(i.e. automatically keep clear of all possible boot ranges of all ARM
vendors) or to even find the extent of the problem.  I think it should be
the vendors' responsibility to standardize on one way to boot that doesn't
suck :P

It turns partitioning basically from straightforward allocation inside
one block into allocation with strange hidden mines to avoid.

Embedded usually ignores the problem, finds a partitioning that works
(with dummy spacer partition if you are lucky--otherwise just random
gap) and everyone just uses that one.  Sigh...

(x86 kinda has standardized a gap at the beginning of the disk until the
second head (seems to be 2048 sectors with 512 Byte/sector nowadays
since the drive doesn't expose valid CHS information anymore), and put
the bootloader there--so that's a 1 MiB gap at the beginning of the
disk)

[1] https://alioth-lists.debian.net/pipermail/parted-devel/2017-December/005152.html

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Use genimage for disk-image creation.

[Mathieu Othacehe]

Hey!

> I’m completely sold to the idea.  :-)

I'm glad you like it!

> Looking at ‘image-ext2.c’ reveals that genimage actually just shells out
> to mke2fs.  Indeed, I discovered that ‘mke2fs -d /my/root’ copies
> /my/root as the image’s root directory.  Likewise, for ISO, it just
> shells out to ‘genisoimage’.
>
> So I think that we could avoid ‘genimage’ altogether and implement
> similar functionality for ext4/ISO in (gnu build disk-image).

Ok! One useful functionality of genimage seems to be that's its capable
of creating complex partitioning layouts. Of course we can re-implement
it in (gnu build disk-image), but I don't know how complex will it be.

Danny, I found the genimage files for each arm/aarch64 boards inside
Buildroot repository. They are a valuable source of information, but
regardless of the decision we take for (gnu build disk-image), we can
find a way to import them and store them as Guile records. 

I guess I'll start a small implementation to have a clearer view of this
topic.

Thank you all for the feedback,

Mathieu

Re: Use genimage for disk-image creation.

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 3485 bytes --]

On 2020-03-29, Danny Milosavljevic wrote:
> Hi Ludo,
>
> On Sun, 29 Mar 2020 16:44:39 +0200
> Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Oh, really?  I’m surprised partitioning causes problems (though I’m
>> not familiar with embedded dev!).
>
> Well, maybe not that bad, but it's pretty bad.
>
> It's because the boot sector for ARM, for some unfathomable reason, is not
> the first sector but has basically a vendor-dependent sector number somewhere
> in the middle of the disk :P
>
> (with the "boot sector" I mean the sector(s) where usually u-boot resides)
>
> So there are all kinds of stupid problems like sometimes you have to reduce
> the size of the GPT partition table (TABLE!) because it would write right over
> the actual boot sector of the platform otherwise.  For that vendor.  Other
> vendors do other silly things.
>
> Parted doesn't really have support for that stuff either, and I while I
> brought it up with them[1], I can't think of a practical way to fix it
> (i.e. automatically keep clear of all possible boot ranges of all ARM
> vendors) or to even find the extent of the problem.  I think it should be
> the vendors' responsibility to standardize on one way to boot that doesn't
> suck :P
>
> It turns partitioning basically from straightforward allocation inside
> one block into allocation with strange hidden mines to avoid.
>
> Embedded usually ignores the problem, finds a partitioning that works
> (with dummy spacer partition if you are lucky--otherwise just random
> gap) and everyone just uses that one.  Sigh...

Someday... but not today. :/

The biggest offset I'm aware of is rockchip platforms, at what I think
is ~16MB:

  http://opensource.rock-chips.com/wiki_Partitions

sunxi/allwinner 64-bit platforms have offsets that conflict with a
standard GPT partition table, though there are rumours of it working
fine as long as you keep the number of partitions below four:

  https://bugs.debian.org/928643

32-bit sunxi/allwinner platforms are generally pretty reasonable, but I
don't recall the issues off the top of my head.

ti/omap/etc platforms tend to fit under 4MB.


I haven't really looked at buildroot at all... but I suspect buildroot
is just a collection of all these criteria applied on a board-by-board
basis, but most of these boards don't actually require specific
partition layout, per se, it's just nice to not clobber the raw offsets
of various parts of the boot process... but creating partitions for each
of those can make installation less error-prone in some cases.

I think the default for most partitioning tools these days is to create
the first partition starting at 2MB, but If the guix partition started
at 16MB and you limited partitions to under four partitions, that should
work for all the platforms I'm aware off of the top of my head... at
least for now...

I seem to also recall an that some disk media (emmc, microSD, maybe some
SSDs) have an erase block granularity of 4MB, and so it'd be ideal if
the first partition started at a multiple of 4MB:

$ lsblk -D

NAME                DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
mmcblk0                    0        4M     3.5G         0

Or alternately, detecting this size and making sure the first partition
starts as a multiple of this value.

I really need to sort some of these issues out in Debian as well, so
happy to carry on the conversation!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Use genimage for disk-image creation.

[Ludovic Courtès]

Hi!

Vagrant Cascadian <vagrant@debian.org> skribis:

> On 2020-03-29, Danny Milosavljevic wrote:
>> Hi Ludo,
>>
>> On Sun, 29 Mar 2020 16:44:39 +0200
>> Ludovic Courtès <ludo@gnu.org> wrote:
>>
>>> Oh, really?  I’m surprised partitioning causes problems (though I’m
>>> not familiar with embedded dev!).
>>
>> Well, maybe not that bad, but it's pretty bad.
>>
>> It's because the boot sector for ARM, for some unfathomable reason, is not
>> the first sector but has basically a vendor-dependent sector number somewhere
>> in the middle of the disk :P

Oh I see, I actually noticed that in (gnu bootloader u-boot) and in
Buildroot, it’s terrrrible.

> I haven't really looked at buildroot at all... but I suspect buildroot
> is just a collection of all these criteria applied on a board-by-board
> basis, but most of these boards don't actually require specific
> partition layout, per se, it's just nice to not clobber the raw offsets
> of various parts of the boot process... but creating partitions for each
> of those can make installation less error-prone in some cases.

Yeah, I guess we could keep importing them like Danny did in the past,
but of course it would be nice to have an automated process to do that.

Ludo’.