Kernel module build error with system image cross-build

Kernel module build error with system image cross-build

[Frank Terbeck]

Good day, good people!

I've  been playing  around with  building guix  images for  a beaglebone
black board that's been catching dust in  my flat. I've been giving it a
go with the template from guix's git repository, like this:

  guix system image --target=arm-linux-gnueabihf ~/src/guix/gnu/system/examples/beaglebone-black.tmpl

After a while, this breaks like this:

  […]
  building /gnu/store/nv461zd7rs6mhns9s3jx4v65ln3jsrv7-linux-modules.drv...
  builder for `/gnu/store/nv461zd7rs6mhns9s3jx4v65ln3jsrv7-linux-modules.drv' failed with exit code 1
  build of /gnu/store/nv461zd7rs6mhns9s3jx4v65ln3jsrv7-linux-modules.drv failed
  View build log at '/var/log/guix/drvs/nv/461zd7rs6mhns9s3jx4v65ln3jsrv7-linux-modules.drv.gz'.
  […]
  guix system: error: build of `/gnu/store/lqwjj7fjf235psw2br5i8y10cm22pq4l-disk-image.drv' failed


The build log looks like this:

  Backtrace:
             5 (primitive-load "/gnu/store/axmhy07daha215gwbqghh39k7ja?")
  In ice-9/eval.scm:
      619:8  4 (_ #f)
     626:19  3 (_ #<directory (guile-user) 7ffff3fd7c80>)
     293:34  2 (_ #(#<directory (guile-user) 7ffff3fd7c80> #<procedu?>))
  In srfi/srfi-1.scm:
     586:17  1 (map1 ("omap_hsmmc" "ahci" "usb-storage" "uas" "usbh?" ?))
  In gnu/build/linux-modules.scm:
      257:5  0 (_)

  gnu/build/linux-modules.scm:257:5: kernel module not found "omap_hsmmc" "/gnu/store/rslz7zlq11wjnvixzfasyvr4b6rv2m7j-linux-libre-5.18.14/lib/modules"


I am not at all familiar with how the kernel (and its modules) build au-
tomation in guix works, so I am  not sure how to approach this.

FWIW, the host system in this case is x86_64, so this is a cross-build.


Any ideas would be appreciated.


Regards, Frank

Re: Kernel module build error with system image cross-build

[Frank Terbeck]

I looked into this over the weekend.

Frank Terbeck wrote:
> […] like this:
>
>   guix system image --target=arm-linux-gnueabihf ~/src/guix/gnu/system/examples/beaglebone-black.tmpl
>
> After a while, this breaks like this:
>
[…]
>   guix system: error: build of `/gnu/store/lqwjj7fjf235psw2br5i8y10cm22pq4l-disk-image.drv' failed
[…]
>   Backtrace:
>              5 (primitive-load "/gnu/store/axmhy07daha215gwbqghh39k7ja?")
>   In ice-9/eval.scm:
>       619:8  4 (_ #f)
>      626:19  3 (_ #<directory (guile-user) 7ffff3fd7c80>)
>      293:34  2 (_ #(#<directory (guile-user) 7ffff3fd7c80> #<procedu?>))
>   In srfi/srfi-1.scm:
>      586:17  1 (map1 ("omap_hsmmc" "ahci" "usb-storage" "uas" "usbh?" ?))
>   In gnu/build/linux-modules.scm:
>       257:5  0 (_)
>
>   gnu/build/linux-modules.scm:257:5: kernel module not found "omap_hsmmc" "/gnu/store/rslz7zlq11wjnvixzfasyvr4b6rv2m7j-linux-libre-5.18.14/lib/modules"

Yeah, so the build-process  here, as far as I can  see, works like this:
Build the kernel, then to build  the initrd, use the kernel installation
directory. The list of modules is just  a list of strings, for which the
system tries  to find corresponding  ‘.ko’ files in the  kernel's module
tree for.  If the module can't be found, this is what you end up with.

The ‘beaglebone-black.tmpl’ file is basically just an ‘operating-system’
specification:

    (operating-system
      ;; …
      ;; This module is required to mount the SD card.
      (initrd-modules (cons "omap_hsmmc" %base-initrd-modules))
      ;; …
      )

There's  also  a   ‘beaglebone-black-installation-os’  specification  in
‘gnu/system/install.scm’.  It uses  the deprecated  ‘#:extra-modules’ to
add ‘omap_hsmmc’ to its list of initrd-modules. So it'll fail similarly.

Odd. So I took a look at the actual kernel build result. When you check
for ‘omap_hsmmc’ you'll notice that it comes up in ‘modules.builtin’, an
output from ‘kbuild’:

    This file lists all modules that are built into the kernel. This is
    used by modprobe to not fail when trying to load something builtin.

This would suggest that  this feature is not built as  a module, but in-
stead is a  fixed part of the  kernel build. And indeed,  looking at the
corresponding ‘.config’ file for the kernel build, you'll find this:

    CONFIG_MMC_OMAP_HS=y

…which is the  Kconfig option, that controls the inclusion  of this fea-
ture. Clearly something changed with time  and this setting is no longer
valid. Fair enough. Take it out and you should be golden, right?

Not quite  — it'll  break again, complaining  about another  module. The
rest  of the  modules is  take from  ‘default-initrd-modules’, which  is
identifier-syntax'd to ‘%base-initrd-modules’.

It's value for the ‘arm-linux-gnueabihf’ target is:

    ahci, usb-storage, uas, usbhid, hid-generic, hid-apple, dm-crypt,
    xts, serpent_generic, wp512, nls_iso8859-1, pata_acpi, pata_atiixp,
    isci, virtio_pci, virtio_balloon, virtio_blk, virtio_net,
    virtio_console, virtio-rng

Of these, the modules that are available as actual modules:

    uas, xts, virtio_pci, virtio_balloon, virtio_blk, virtio_net

Features that are builtin, and thus not available as a ‘.ko’ file:

    ahci, usb-storage, usbhid, hid-generic, nls_iso8859-1,
    virtio_console

Unavailable features:

    hid-apple, dm-crypt, serpent_generic, wp512, pata_acpi, pata_atiixp,
    isci, virtio-rng

With all that in mind, I used this specification; mostly based on what's
in ‘beaglebone-black.tmpl’:

#+begin_src scheme
(use-modules (srfi srfi-1)
             (gnu)
             (gnu bootloader u-boot))

(use-service-modules networking)
(use-package-modules bootloaders screen ssh)

(operating-system
  (host-name "bbb-test")
  (timezone "Europe/Berlin")
  (locale "en_GB.utf8")
  (bootloader (bootloader-configuration
               (bootloader u-boot-beaglebone-black-bootloader)
               (targets '("/dev/mmcblk1"))))
  (initrd-modules
    (map symbol->string
         '(uas xts virtio_pci virtio_balloon virtio_blk virtio_net)))
  (file-systems (cons (file-system (device (file-system-label "my-root"))
                                   (mount-point "/")
                                   (type "ext4"))
                      %base-file-systems))
  (users (cons (user-account (name "ft")
                             (home-directory "/home/ft")
                             (group "users")
                             (supplementary-groups '("wheel" "audio" "video")))
               %base-user-accounts))
  (packages (cons* screen openssh %base-packages))
  (services (cons* (service dhcp-client-service-type)
                   (agetty-service (agetty-configuration (extra-options '("-L"))
                                                         (baud-rate "115200")
                                                         (term "vt100")
                                                         (tty "ttyO0")))
                   %base-services)))
#+end_src

This builds.

Not sure of it boots, because of  course none of my micro-sd cards work.
I'm building  a VM image now  and will try to  see if QEMU can  boot the
thing.


Regards, Frank
-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Kernel module build error with system image cross-build

[pelzflorian (Florian Pelz)]

Hello Frank, sorry for the late reply.  HDMI stays black, but I can’t
find the cables for my UART adapter, so I cannot properly test.  I think
you are doing it right, but if what used to work does not, perhaps an
easier way would be to copy the rock64.scm image, and adapt that to use
the Beaglebone Black’s u-boot and the kernel linux-libre-arm-generic
instead of linux-libre-arm64-generic, then pass this file to guix system
image.  But perhaps this still is the wrong kernel and you need an older
kernel, either as an old linux-libre-arm-generic-4.19 kernel or an old
kernel via a Guix inferior.

Regards,
Florian

Re: Kernel module build error with system image cross-build

[Frank Terbeck]

pelzflorian@pelzflorian.de wrote:
> Hello Frank, sorry for the late reply.

Hey Florian,  no worries!  I'm only  working on  this when  time permits
anyway. It  is cool when  things aren't complete monologues,  though. So
thanks for chiming in. :)

It's interesting to see how Guix does these things.


> HDMI stays black, but I can’t
> find the cables for my UART adapter, so I cannot properly test.  I think

Ah, so you tried, nice! I couldn't yet… Maybe this weekend.


> you are doing it right, but if what used to work does not, perhaps an
> easier way would be to copy the rock64.scm image, and adapt that to use
> the Beaglebone Black’s u-boot and the kernel linux-libre-arm-generic
> instead of linux-libre-arm64-generic, then pass this file to guix system
> image.

Certainly. I didn't even know gnu/system/images was a place to look. :)

I'm  still finding  out about  Guix's structure  and API.  Apart from  a
couple of  package definitions, I  didn't do a  lot yet. Things  in this
field do  feel a little  all over to  place, though. There's  these tem-
plates, then  there's gnu/system/install.scm  (which also has  a rock64-
installation-os definition) and also this  images place. Maybe there's a
point to it, that I'm not seeing.  Or maybe it's just a lack of manpower
to keep stuff up  to date when the rest of the project  moves along at a
swift pace.


> But perhaps this still is the wrong kernel and you need an older
> kernel, either as an old linux-libre-arm-generic-4.19 kernel or an old
> kernel via a Guix inferior.

I think the interface into the kernel  data is lacking. It might be use-
ful to have an interface to the kernel's ‘.config’, as well as meta data
files such at the mentioned ‘modules.builtin’.

The fact that the beaglebone  specs reference "omap_hsmmc" is correct on
its face, because that driver is  indeed required. If it's builtin, then
not finding the file is to be expected and not an issue.

With better  introspection to  the kernel's  build configuration,  an OS
specification could easier react to upstream changes, I think.

Not sure if that sort of discussion isn't more appropriate for -devel.


Regards, Frank
-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Kernel module build error with system image cross-build

[pelzflorian (Florian Pelz)]

Frank Terbeck <ft@bewatermyfriend.org> writes:
> pelzflorian@pelzflorian.de wrote:
>> HDMI stays black, but I can’t
>> find the cables for my UART adapter, so I cannot properly test.  I think
>
> Ah, so you tried, nice! I couldn't yet… Maybe this weekend.

Yes, although by now I noticed HDMI stays black for my Beaglebone Black
on the official Debian 8, 9, 10 images too.  But unlike your image, on
the official images the keyboard is powered so that I can reboot with
Ctrl+Alt+Del and that works because the BBB LEDs turn off and on again.
Also I can ping it.

(I am on the lookout for what to run my server on; I guess the dusty BBB
is out.  Back in the day Debian 8 worked and Debian 10 worked after
changing the .dtb file
<https://elinux.org/BeagleBoneBlack_Stock_Debian_from_External_HD>.)


>  do  feel a little  all over to  place, though. There's  these tem-
> plates, then  there's gnu/system/install.scm  (which also has  a rock64-
> installation-os definition) and also this  images place.

Images are more constrained in that they cannot be used for guix system
init.

>> But perhaps this still is the wrong kernel and you need an older
>> kernel, either as an old linux-libre-arm-generic-4.19 kernel or an old
>> kernel via a Guix inferior.

Using a custom u-boot-beaglebone-black-bootloader changed to use an
inferior of Guix commit 6b99afeef89233b71d113a63cf04a6b4b49a4679 for
u-boot did not boot either and only gets hot and cannot be pinged.  I
have not yet tried older u-boots or kernels.


> I think the interface into the kernel  data is lacking. It might be use-
> ful to have an interface to the kernel's ‘.config’, as well as meta data
> files such at the mentioned ‘modules.builtin’.
> […]
> Not sure if that sort of discussion isn't more appropriate for -devel.

If one of us finds a working beaglebone-black.tmpl or makes progress, or
has a proposal for kernel code/docs, then it certainly belongs in guix-devel
or guix-patches.  Note that `info guix-cookbook` has more info on
building a custom kernel; maybe it is reasonably recent.

Regards,
Florian