Building installation image for ROCK64

Building installation image for ROCK64

[Simon South]

Has anyone successfully built an installation image for a PINE64 ROCK64
ARM SBC?

There's a definition for it in gnu/system/install.scm, but building the
image with

  guix system disk-image --system=aarch64-linux \
    -e "(@ (gnu system install) rock64-installation-os)"

and writing it to a microSD card fails to boot completely as the root
filesystem can't be mounted:

  GC Warning: pthread_getattr_np or pthread_attr_getstack failed for main thread
  GC Warning: Couldn't read /proc/stat
  Welcome, this is GNU's early boot Guile.
  Use '--repl' for an initrd REPL.

  loading kernel modules...
  waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
  waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
  (...)
  waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
  ERROR: In procedure scm-error:
  failed to resolve partition "416bf41b-f6b1-2062-6e00-1979416bf41b"
  
(I've checked and the GUID above does match the root partition.) I
understand this often means a necessary kernel module is missing from
initrd, but rebuilding the image using the definition below (mostly
copied from install.scm) that explicitly includes the Rockchip MMC
driver produces the same non-functioning result.

Do you know the magic incantation necessary to produce a bootable image
for the ROCK64?

Alternatively, how might I proceed in diagnosing the issue here?

  (use-modules (gnu system install)
               (gnu system linux-initrd)
               (gnu bootloader)
               (gnu bootloader u-boot)
               (gnu packages linux))

  (operating-system
   (inherit installation-os)
   (bootloader (bootloader-configuration
                (bootloader u-boot-rock64-rk3328-bootloader)
                (target "/dev/mmcblk0")))
   (kernel linux-libre)
   (kernel-arguments
    (cons "console=ttyS2"
          (operating-system-user-kernel-arguments installation-os)))
   (initrd-modules (append '("dw_mmc" "dw_mmc-rockchip") %base-initrd-modules)))

-- 
Simon South
simon@simonsouth.net

Re: Building installation image for ROCK64

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2749 bytes --]

On 2020-04-11, Simon South wrote:
> Has anyone successfully built an installation image for a PINE64 ROCK64
> ARM SBC?

I haven't tested it yet...

>
> There's a definition for it in gnu/system/install.scm, but building the
> image with
>
>   guix system disk-image --system=aarch64-linux \
>     -e "(@ (gnu system install) rock64-installation-os)"
>
> and writing it to a microSD card fails to boot completely as the root
> filesystem can't be mounted:

Sounds like the bootloader actually works, so not *completely* ! :)

>   GC Warning: pthread_getattr_np or pthread_attr_getstack failed for main thread
>   GC Warning: Couldn't read /proc/stat
>   Welcome, this is GNU's early boot Guile.
>   Use '--repl' for an initrd REPL.
>
>   loading kernel modules...
>   waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
>   waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
>   (...)
>   waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
>   ERROR: In procedure scm-error:
>   failed to resolve partition "416bf41b-f6b1-2062-6e00-1979416bf41b"
>   
> (I've checked and the GUID above does match the root partition.) I
> understand this often means a necessary kernel module is missing from
> initrd
...
>    (kernel linux-libre)
>    (kernel-arguments
>     (cons "console=ttyS2"
>           (operating-system-user-kernel-arguments installation-os)))
>    (initrd-modules (append '("dw_mmc" "dw_mmc-rockchip") %base-initrd-modules)))

You're probably missing some voltage regulators and a handful of other
things...

This might avoid playing whack-a-mole with arbitrarily complex sets of drivers:

   (kernel linux-libre-arm64-generic)
   (initrd-modules '())

I've been using the linux-libre-arm*-generic* kernels for precisely this
reason; they tend to have the drivers built-in and the configs are
maintained upstream reasonably well. Also happen to be the first
platforms supporting linux-libre 5.6.x in guix master... :)


The *-generic kernels won't work for complex setups (e.g. encrypted
rootfs), so at some point it might be good to try and figure out a
helper or something to parse one or more relevent device-tree and kernel
config to generate the right set of modules. So I'd lean towards some
way to include more modules in the initrd by default, while handling
that some might be built-ins or not present for certain architectures,
etc...


I have a single rootfs that boots on both the pinebook and pinebook-pro
(switching out the boot firmware only), which are pretty different under
the hood, so it would be nice to not have to make an image that was
device-specific.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Building installation image for ROCK64

[Simon South]

Vagrant Cascadian <vagrant@debian.org> writes:
> This might avoid playing whack-a-mole with arbitrarily complex sets of drivers:
>
>    (kernel linux-libre-arm64-generic)
>    (initrd-modules '())

Thanks, Vagrant. That definitely got me further, though the machine
still fails to finish booting:

  [    0.243248] rockchip-pinctrl pinctrl: pin gpio0-2 already requested by vcc-host-5v-regulator; cannot claim for vcc-host1-5v-regulator
  [    0.243264] rockchip-pinctrl pinctrl: pin-2 (vcc-host1-5v-regulator) status -22
  [    0.243279] rockchip-pinctrl pinctrl: could not request pin 2 (gpio0-2) from group usb20-host-drv  on device rockchip-pinctrl
  [    0.243291] reg-fixed-voltage vcc-host1-5v-regulator: Error applying setting, reverse things back
  GC Warning: pthread_getattr_np or pthread_attr_getstack failed for main thread
  GC Warning: Couldn't read /proc/stat
  Welcome, this is GNU's early boot Guile.
  Use '--repl' for an initrd REPL.

  loading kernel modules...
  loading '/gnu/store/6a4pyi34awj0jkd6ipl39dylj675ipxm-system/boot'...
  ERROR: In procedure open-file:
  In procedure open-file: No such file or directory: "/var/run/utmpx"

  Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
  GNU Guile 2.2.6
  Copyright (C) 1995-2019 Free Software Foundation, Inc.

I assume the missing file is the actual reason for the failure and
whatever's going on with the voltage-regulator drivers is unimportant
for now. Any chance something in the output above rings a bell?

I'll start diagnosing from here---it's probably time to write a real
"operating-system" definition for this machine and see what a newly
generated image contains.

-- 
Simon South
simon@simonsouth.net

Re: Building installation image for ROCK64

[Simon South]

Vagrant,

Thanks to help from you, Pierre Langlois and a few others on IRC I've
had Guix System running on my ROCK64 for a while now. I thought I'd
follow up with the configuration I used and some notes in case it's
helpful to someone else down the road:

I started with an existing GNU/Linux distribution (Armbian Bionic,
though almost any should work fine) installed on my ROCK64's internal
eMMC module, to which I added Guix using the installer script linked
from chapter 2 of the Guix manual.

From there, log in to the ROCK64 as the superuser. Using fdisk, prepare
a microSD card (/dev/mmcblk0) with a GPT partition table and a single
partition beginning at sector 2,048. Format the partition
(/dev/mmcblk0p1) as ext4, then mount it at /mnt.

Create /mnt/etc and write a configuration file like the one below
(perhaps changing the timezone and locale and adding a user account) to
/mnt/etc/config.scm.

Then, making sure the Guix build daemon is running, have Guix populate
the microSD card with

  guix system init /mnt/etc/config.scm /mnt

Once this completes the card will contain a bootable Guix System
image. Power off the ROCK64, place a jumper over the pins behind the
power LED to disable the eMMC clock, and power on the machine again.  It
should boot from the card into Guix System.

From there, the OS can be installed to the eMMC module through a
complicated dance: Using another computer connected to the ROCK64 via
its serial interface, hit Enter during startup to reach the U-Boot
prompt, then carefully (!) remove the jumper from the ROCK64 to
re-enable its eMMC module and enter "boot" to finish booting into
Guix. Edit /etc/config.scm to replace "/dev/mmcblk0" with "/dev/mmcblk1"
(the eMMC module), then repeat the steps above (making the same
substitution) to prepare the module and install a fresh copy of the OS.

The configuration file below is fairly minimal but adds a DHCP client
and OpenSSH server to the base services so the ROCK64 can be accessed
easily once it's connected to a network.

Importantly, it also adds an NTP service to set the date and time at
startup. The ROCK64 has no RTC battery and defaults to 1 January 2016 as
the date at each boot, which causes "guix pull" to fail with an
SSL-certificate error until the clock is set properly. (Note OpenNTPD
itself ocassionally fails to set the clock and it may be necessary to
run "sudo herd restart ntpd" manually after booting.)

Thanks again to you and everyone else who helped me reach this point.

----------

(use-modules (gnu)
	     (gnu bootloader u-boot)
	     (gnu system nss))
(use-package-modules certs linux ntp)
(use-service-modules networking ssh)

(operating-system
 (host-name "rock64")
 (timezone "America/Toronto")
 (locale "en_CA.utf8")

 (bootloader (bootloader-configuration
	      (bootloader u-boot-rock64-rk3328-bootloader)
	      (target "/dev/mmcblk0")))

 (initrd-modules '())

 (kernel linux-libre-arm64-generic)
 (kernel-arguments '("console=ttyS2,1500000"))

 (file-systems
  (cons* (file-system
	  (mount-point "/")
	  (device "/dev/mmcblk0p1")
	  (type "ext4"))
	 %base-file-systems))

 (packages (append (list nss-certs) %base-packages))
 
 (services (append (list (service dhcp-client-service-type)
			 (service openntpd-service-type
			  (openntpd-configuration
			   (allow-large-adjustment? #t)))
			 (service openssh-service-type))
			 %base-services)))

-- 
Simon South
simon@simonsouth.net

Re: Building installation image for ROCK64

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On 2020-04-26, Simon South wrote:
> From there, log in to the ROCK64 as the superuser. Using fdisk, prepare
> a microSD card (/dev/mmcblk0) with a GPT partition table and a single
> partition beginning at sector 2,048.

With your current layout, parts of the bootloader may be written to the
same offsets as files in your first partition... you may have gotten
lucky for now, but it's only a matter of time till something breaks your
bootloader!

You really want to have the loader1 (start sector 64, 2.5MB size) and
loader2 (start sector 16384, 4MB size) partitions listed in:

  http://opensource.rock-chips.com/wiki_Partitions

gptfdisk/gdisk should be able to write such a partition table (at least
the loader2 partition, some tools refuse to create the loader1 partition
starting at sector 64).


Glad you made good progress! It would be nice to eventually be able to
create installer images for aarch64/armhf... there's been a lot of
progress on being able to create cross-compiled installation images, so
the day may not be so far off!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Building installation image for ROCK64

[Simon South]

Vagrant Cascadian <vagrant@debian.org> writes:
> With your current layout, parts of the bootloader may be written to the
> same offsets as files in your first partition...

Yes, my mistake. Thanks for pointing that out.

> You really want to have the loader1 (start sector 64, 2.5MB size) and
> loader2 (start sector 16384, 4MB size) partitions...

I'm not sure how literally you meant this to be interpreted, but after a
bit of experimentation it seems the most sensible arrangement for now is
just to have a single partition starting at sector 32,768 for the root
filesystem. This is because

- If real partitions are created for the bootloader stages and the
  trusted firmware, U-Boot will fail to start the OS (with "Unrecognized
  filesystem type") when it scans for bootable partitions. (It probably
  ought to just skip over partitions without a recognizable filesystem,
  but it doesn't seem to behave that way.)

- It seems Guix System does not yet support having /boot on a separate
  partition and will fail at startup if the store isn't available on the
  same filesystem as extlinux.conf. Consequently reserving 112 MB for a
  separate boot partition accomplishes nothing.

At least this way the root filesystem is safe from being overwritten by
the bootloader, and as Guix's support for multiple partitions improves
over time it'll be possible to more closely follow Rockchip's
conventions.

> It would be nice to eventually be able to create installer images for
> aarch64/armhf...

Yes, absolutely. In the meantime just making available a
minimal-but-complete image for writing to a microSD card would be a big
help to people looking to get started quickly with Guix on the ROCK64.

-- 
Simon South
simon@simonsouth.net

Re: Building installation image for ROCK64

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2397 bytes --]

On 2020-04-27, Simon South wrote:
> Vagrant Cascadian <vagrant@debian.org> writes:
>> With your current layout, parts of the bootloader may be written to the
>> same offsets as files in your first partition...
>
> Yes, my mistake. Thanks for pointing that out.
>
>> You really want to have the loader1 (start sector 64, 2.5MB size) and
>> loader2 (start sector 16384, 4MB size) partitions...
>
> I'm not sure how literally you meant this to be interpreted, but after a
> bit of experimentation it seems the most sensible arrangement for now is
> just to have a single partition starting at sector 32,768 for the root
> filesystem. This is because

That also should work fine.


> - If real partitions are created for the bootloader stages and the
>   trusted firmware, U-Boot will fail to start the OS (with "Unrecognized
>   filesystem type") when it scans for bootable partitions. (It probably
>   ought to just skip over partitions without a recognizable filesystem,
>   but it doesn't seem to behave that way.)

It's possible to mark the partition as bootable (I think "ESP Boot" or
something like that).


> - It seems Guix System does not yet support having /boot on a separate
>   partition and will fail at startup if the store isn't available on the
>   same filesystem as extlinux.conf. Consequently reserving 112 MB for a
>   separate boot partition accomplishes nothing.

Right; I wouldn't suggest using such a tiny boot partition anyways (I
think they suggest a 112MB /boot ?).


> At least this way the root filesystem is safe from being overwritten by
> the bootloader, and as Guix's support for multiple partitions improves
> over time it'll be possible to more closely follow Rockchip's
> conventions.

Yeah, anything after sector 32768 is, in my opinion, fair game to do
whatever you want with. :)


>> It would be nice to eventually be able to create installer images for
>> aarch64/armhf...
>
> Yes, absolutely. In the meantime just making available a
> minimal-but-complete image for writing to a microSD card would be a big
> help to people looking to get started quickly with Guix on the ROCK64.

I'm tempted to make armhf and aarch64 one-off images using
linux-image-arm64-generic and linux-image-arm-generic kernels, and
include instructions for updating the bootloader, to support an
arbitrary number of different systems... this should be doable.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Building installation image for ROCK64

[Pierre Langlois]

Hi Simon,

Simon South writes:

> Has anyone successfully built an installation image for a PINE64 ROCK64
> ARM SBC?

I managed to setup a ROCKPRO64 a few months ago (rk3399 chipset instead
of rk3328), maybe I can help :-).

>
> There's a definition for it in gnu/system/install.scm, but building the
> image with
>
>   guix system disk-image --system=aarch64-linux \
>     -e "(@ (gnu system install) rock64-installation-os)"

You probably noticed this takes a *long* time to run, making it quite
tedious to test changes (I believe this is being worked on though).

In the meanwhile, what you can do instead is setup the SD card manually,
say using fdisk & mkfs.ext4, and then use the 'guix system init' command
to install guix on it:

  $ mount /dev/mmcblk0 /mnt
  $ guix system init my-config.scm /mnt

That's considerably faster and should take care of installing guix and
the bootloader in the right place.

>
> and writing it to a microSD card fails to boot completely as the root
> filesystem can't be mounted:
>
>   GC Warning: pthread_getattr_np or pthread_attr_getstack failed for main thread
>   GC Warning: Couldn't read /proc/stat
>   Welcome, this is GNU's early boot Guile.
>   Use '--repl' for an initrd REPL.
>
>   loading kernel modules...
>   waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
>   waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
>   (...)
>   waiting for partition '416bf41b-f6b1-2062-6e00-1979416bf41b' to appear...
>   ERROR: In procedure scm-error:
>   failed to resolve partition "416bf41b-f6b1-2062-6e00-1979416bf41b"
>
> (I've checked and the GUID above does match the root partition.) I
> understand this often means a necessary kernel module is missing from
> initrd, but rebuilding the image using the definition below (mostly
> copied from install.scm) that explicitly includes the Rockchip MMC
> driver produces the same non-functioning result.
>
> Do you know the magic incantation necessary to produce a bootable image
> for the ROCK64?
>
> Alternatively, how might I proceed in diagnosing the issue here?
>
>   (use-modules (gnu system install)
>                (gnu system linux-initrd)
>                (gnu bootloader)
>                (gnu bootloader u-boot)
>                (gnu packages linux))
>
>   (operating-system
>    (inherit installation-os)
>    (bootloader (bootloader-configuration
>                 (bootloader u-boot-rock64-rk3328-bootloader)
>                 (target "/dev/mmcblk0")))
>    (kernel linux-libre)
>    (kernel-arguments
>     (cons "console=ttyS2"
>           (operating-system-user-kernel-arguments installation-os)))

To help debug this, you might want to remove the "quiet" argument so you
see some output from the kernel, I think it's added by default in
installation-os. Just setting (kernel-arguments '("consolte=ttyS2"))
should do.

>    (initrd-modules (append '("dw_mmc" "dw_mmc-rockchip") %base-initrd-modules)))

As Vagrant pointed out, you're probably missing some more modules. On
the rockpro64 I have the following:

    (initrd-modules
     (cons* "dw_mmc-rockchip"
            "phy-rockchip-emmc"
            "sdhci-of-arasan" ;; I believe this one is for rk3399 only,
                              ;; so won't help you.
            %base-initrd-modules))

It was quite some back-and-forth to figure out what was needed. What I
did was to boot a different distro, I think it was Arch at the time, and
look at the dmesg output to see what drivers were loading that I didn't
see with guix.

Hope this helps!

Thanks,
Pierre

Re: Building installation image for ROCK64

[Pierre Langlois]

Pierre Langlois writes:

> Hi Simon,
>
> Simon South writes:
>
>> Has anyone successfully built an installation image for a PINE64 ROCK64
>> ARM SBC?
>
> I managed to setup a ROCKPRO64 a few months ago (rk3399 chipset instead
> of rk3328), maybe I can help :-).
>
>>
>> There's a definition for it in gnu/system/install.scm, but building the
>> image with
>>
>>   guix system disk-image --system=aarch64-linux \
>>     -e "(@ (gnu system install) rock64-installation-os)"
>
> You probably noticed this takes a *long* time to run, making it quite
> tedious to test changes (I believe this is being worked on though).
>
> In the meanwhile, what you can do instead is setup the SD card manually,
> say using fdisk & mkfs.ext4, and then use the 'guix system init' command
> to install guix on it:
>
>   $ mount /dev/mmcblk0 /mnt
>   $ guix system init my-config.scm /mnt

Oh, also, when doing this make sure my-config.scm has the correct target
in the bootloader entry, to prevent accidentally writing the bootloader
on the wrong SD card (from personal experience breaking the host system
that lived on that other SD card :-D).

Re: Building installation image for ROCK64

[Simon South]

Pierre Langlois <pierre.langlois@gmx.com> writes:
> You probably noticed this takes a *long* time to run, making it quite
> tedious to test changes (I believe this is being worked on though).
>
> In the meanwhile, what you can do instead is setup the SD card
> manually...

Thanks, this should be a BIG help. It takes around three hours and
twenty minutes to build a new image on the ROCK64 itself which, as you
say, slows down testing considerably. (I assume the main reason is that
KVM support is explicitly disabled for aarch64 in gnu/system/vm.scm, but
that's a separate issue.)

> It was quite some back-and-forth to figure out what was needed. What I
> did was to boot a different distro, I think it was Arch at the time, and
> look at the dmesg output to see what drivers were loading that I didn't
> see with guix.

This sounds like a good plan.

Thanks very much for the tips, Pierre. I'll give them all a shot and
report the results.

-- 
Simon South
simon@simonsouth.net