Re: Supporting *multiple* bootloaders for arm64 on a single install?

[Vagrant Cascadian <vagrant@debian.org>]

[-- Attachment #1: Type: text/plain, Size: 10441 bytes --]

On 2021-06-19, Danny Milosavljevic wrote:
> On Sun, 06 Jun 2021 14:38:27 -0700
> Vagrant Cascadian <vagrant@debian.org> wrote:
>
>> So, I've managed to get a single image that supports booting both the
>> Pinebook and Pinebook Pro reasonably well! I can pop the microSD card
>> out of one and put it into the other, and it boots!
>
> Awesome!

Thanks! and thanks for stirring up this quiet topic again :)

>> The only problem with my single image for multiple pinebooks variants is
>> it requires manually installing u-boot to different offsets for Pinebook
>> Pro (e.g. idbloader.img at sector 2112 rather than sector 64), 
>
>>as parts of the Pinebook bootloader are installed at overlapping offsets.
>
> Uhhh?  Doesn't sound possible to me then to use the same image for both?
> I'm probably missing something.

Different parts of the image can be written to different offsets on the
boot media in a way that doesn't conflict. The boot ROM on allwinner and
rockchip have a series of offsets they check for compatible boot code,
and the offsets are different enough that you can have a pairing of
allwinner and rockchip boards installed on the same boot media. The pair
that makes "obvious" sense is the pinebook and pinebook-pro, though
nearly any pairing of allwinner and rockchip boards would likely work.


>> To support all of the above, I would need three separate bootloader
>> instances... one for Pinebook, one for Pinebook Pro, and lastly a
>> grub-efi bootloader.
>
> Stefan wrote a Guix chain bootloader that is in master--but it's meant
> to be only used for bootloaders where there is a primary "bare-metal-loaded"
> bootloader and the others are chain-loaded one-after-another from ONE filesystem
> (for example Raspberry Pi and/or EFI bootloaders).
>
> (It's currently used in order to use an EFI bootloader hosted on NFS--which
> is an awesome way to manage embedded boards)
>
> The chain bootloader itself is one Guix bootloader.
>
> I advise you to search for mails by Stefan on the guix-devel mailing list--those
> are very illuminating.

Sounds possibly useful for this, yes....


>> Installing u-boot-pinebook uses:
>
>> (define install-allwinner64-u-boot
>>   #~(lambda (bootloader root-index image)
>>       (let ((spl (string-append bootloader "/libexec/u-boot-sunxi-with-spl.bin"))
>>             (u-boot (string-append bootloader "/libexec/u-boot-sunxi-with-spl.fit.itb")))
>>         (write-file-on-device spl (stat:size (stat spl))
>>                               image (* 8 1024))
>>         (write-file-on-device u-boot (stat:size (stat u-boot))
>>                               image (* 40 1024)))))
>> 
>> Installing u-boot-pinebook-pro-rk3399 uses:
>> 
>> (define install-rockpro64-rk3399-u-boot
>>   #~(lambda (bootloader root-index image)
>
>     #~(lambda* (bootloader root-index image #:key idb-destination-offset)
>
>>       (let ((idb (string-append bootloader "/libexec/idbloader.img"))
>>             (u-boot (string-append bootloader "/libexec/u-boot.itb")))
>>         (write-file-on-device idb (stat:size (stat idb))
>>                               image (* 64 512))
>
>
>                                 image (or idb-destination-offset (* 64 512)))
>
>>         (write-file-on-device u-boot (stat:size (stat u-boot))
>>                               image (* 16384 512)))))
>> 
>> (define install-pinebook-pro-rk3399-u-boot install-rockpro64-rk3399-u-boot)

Thanks! So I would need to add this in the bootloader section of
config.scm and add the optional idb-destination-offset argument to all
the intermediary functions...


>> But now I need to figure out how to pass a non-default offset for the
>> "idb" part for rockchip platforms.
>> 
>> In a system config.scm, you'd define it like so:
>> 
>>   (bootloader (bootloader-configuration
>>                (bootloader u-boot-pinebook-pro-rk3399-bootloader)
>>                (target "/dev/mmcblk0")))  
>> 
>> u-boot-pinebook-pro-rk3399-bootloader is defined in
>> gnu/bootloader/u-boot.scm, which inherits from u-boot-bootloader, which
>> inherits from extlinux-bootloader in gnu/bootloader/extlinux.scm...
>> 
>> And somewhere along the way I've lost track of how we get to
>> install-pinebook-pro-rk3399-u-boot...
>
> It's in the "bootloader" record, field "disk-image-installer".
>
> If you search for "bootloader-disk-image-installer" in the Guix source code,
> you find the (two) callers of it.
>

>> Is it possible to definte multiple bootloaders [in the same operating-system] currently, 
>
> No--and I don't think that would have any advantages.
>
> If the thing is not a chain then Guix doesn't need to logically split the
> bootloaders in the first place.
>
> Ok: stage1 -> stage2 -> stage3 represented as (chain-bootloaders stage1 stage2 stage3).
>
> Not ok: stage1-arch1 stage1-arch2 represented as two different bootloaders
> that both are specified in the same operating-system inside the same list.
>
> Instead, I think it's fine to just model that as one "weird" bootloader
> (arm64-pinebook-generic-bootloader)--especially if both use u-boot anyway and also
> especially if they reuse some of their parts anyway (it's gonna have weird
> interdependencies in any case--so any apparent modularity would just be a ruse).

Yeah, the two u-boot (pinebook, pinebook-pro-rk3399) used are entirely
independent, neither of those are part of a single chain. But either
u-boot variant *could* load grub-efi (and does look for potentialy EFI
images if there's no extlinux.conf or boot.scr) as part of a bootloader
chain...


>> or if not,
>> what would need to change to be able to support that? Where would one
>> pass non-default offsets for a given platform?
>
> Could you elaborate on what you mean?  Pass where?  On the Guix command line?

In config.scm. Sounds like making a specific
arm64-pinebook-generic-bootloader like you suggested above might be an
option, although it would be nice to be able to generalize it enough so
that you could pick semi-arbitrary u-boot combinations (as long as
they're compatible).


> If you mean the disk-image-installer, that's just a procedure (which is run
> build-side).  You can make it do whatever you want--including hardcoding weird
> offsets.  I'm pretty sure it gets the entire image (one could potentially mangle)
> as a parameter anyway.

I've never successfully or knowingly used the disk-image-installer, as
it doesn't support native builds. :/

Mostly, I've been doing "guix system init" and "guix system
reconfigure", using the u-boot-pinebook-bootloader in config.scm since
the default offsets work, and then manually installing the appropriate
u-boot-pinebook-rk3399 bits, due to needing to use non-default offsets.


>> Strictly speaking, the extlinux.conf generation would be optional for an
>> EFI generated image(as u-boot can load grub-efi), although at the moment
>> I'd prefer using extlinux.conf if available as it makes for more
>> consistent matching of device-tree/.dtb file with the running kernel...
>
> If you do want to chainload grub-efi eventually, that's supported in Guix master,
> see "efi-bootloader-chain".  It's meant for the end user to be able to use.

I'll look into it, thanks!


>> A related idea would be to generate an EFI bootable image with
>> sufficient emtpy space before the first partition (16MB) and then one
>> could manually install the appropriate u-boot, maybe with some helper
>> scripts to avoid having to do it completely manually...
>
> Yeah, that's possible right now.  After all, you don't have to load the generated
> guix system disk image on bare metal.  Just boot it in qemu and install u-boot
> there for example.

No need to boot into qemu to install u-boot, it's just typically write a
couple binaries to a specific offset in the resulting image...


> Or even edit the image manually by using dd.  (unfortunately,
> on almost every ARM system I set up Guix on so far, one or both of those were
> necessary)

Curious. I've had good luck with:

  (bootloader (bootloader-configuration
                (bootloader u-boot-pinebook-pro-rk3399-bootloader)
                (target "/dev/mmcblk0")))  

Occasionally having to do weird things like passing /dev/sda instead if
installing to a USB adapter for microSD or something.


> That said, it's nice to have officially supported ARM systems where you can just
> generate a disk image and it will boot without having to do weird extra stuff
> outside Guix.

That is the goal!


> (In general, I want to reiterate that the buildroot project already has genimage
> scripts for almost all ARM boards--and Guix uses genimage anyway!  It makes a lot
> of sense to write an importer for those instead of reinventing the wheel.  You
> really do not want to develop & maintain u-boot installation recipes for all
> 34432 ARM variants out there yourself--just use the recipes already developed
> instead)

Admittedly, I haven't looked into buildroot's genimage scripts beyond
just a cursory glance... though have been meaning to even since you
mentioned it what now seems ages ago. Maybe an importer would make a lot
of sense, though it would likely loose some flexibility along the way
(like my mad scientist schemes to install two u-boots on the same
system).

I've written some simple shell scripts for allwinner and rockchip
systems for Debian that basically just dd various bits to offsets.

And of course, I have looked at the code to do essentially the same
thing in guix but in guile... I do wonder if the u-boot-FOO to
install-u-boot-FOO chain could be refactored to remove a few
intermediate steps, which is what I was hinting at in my earlier mail.

When using upstream u-boot, it is a relatively small set of
instructions; e.g. allwinner uses one set of instructions, rockchip uses
another set of instructions, ti-family processors another, imx
processors, and maybe a 32-bit variant for some of them... so ~6
standard installation sets covers hundreds or even thousands of boards,
with a few outliers that differ. Most other platforms I'm aware of
require non-free bits so are out of scope for guix.


And thanks for reading thus far my rambling post about niche corner
cases that are actually useful! :)


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]