Re: ABI mismatch on boot on arm32 system

[Christoph Buck <dev@icepic.de>]

Hi!

I played around a little bit more and i can indeed now successfully
boot. Instead of using cross-compilation (cli option
`--target=arm-linux-gnueabihf`) i created a build using qemu emulation
(cli option `--system=armhf-linux`). This takes ages to build, but the
resulting images is bootable without abi error. Unfortunatly this is not
a real fix because it is too slow to be a practical workaround (at least
for me).

I digged a little deeper and this is what i found out so far. In case i
am running off in a totally wrong direction, someone with more clue than
me should please stop me ;)

I think something goes wrong during crosscompilation of the guile
modules in package `module-import-compiled`. The abi error is thrown
early on boot in the `initrd.cpio.gz` ramdisk. I extracted and
decompressed the ramdisk from both builds (crosscompilation and qemu)
which contain the `module-import-compiled` guile modules. I would expect
that the *.go files from the `module-import-compiled` package of both
ramdisks are binary identical but they have different md5sums. Lets take
for example `file-systems.go`, which cause the abi error.

--8<---------------cut here---------------start------------->8---
local@host:crosscompilation-initrd/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system$
md5sum file-systems.go
7839e9c7a0c7c6c8d9ea45566ab9f61e  file-systems.go
--8<---------------cut here---------------end--------------->8---
vs
--8<---------------cut here---------------start------------->8---

local@host:qemu-initrd/gnu/store/hvgj80xqf70mvx460pnvwmi87kqqn2bj-module-import-compiled/gnu/system$
md5sum file-systems.go
a43a7e939ae9d0cc1ce30726cb51d6d4  file-systems.go
--8<---------------cut here---------------end--------------->8---

Additional it looks like different symbols are exported depending if
cross-compilation or qemu was used. This is at least what `readelf -s file-system.go`
reports. I naively thought these files should be identical.

Additional i saw these strange errors in the build log during
crosscompilation

--8<---------------cut here---------------start------------->8---
;;; WARNING: loading compiled file
/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/build/file-systems.go
failed:
;;; In procedure load-thunk-from-memory: ELF file does not have native word size
;;; WARNING: loading compiled file
/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system/uuid.go
failed:
;;; In procedure load-thunk-from-memory: ELF file does not have native word size
;;; WARNING: loading compiled file
/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system/file-systems.go
failed:
;;; In procedure load-thunk-from-memory: ELF file does not have native word size
--8<---------------cut here---------------end--------------->8---

This also looks suspicious. These stem from the `check_elf_header`
function in guile. Guile warns that the class type in the elf header is
32bits if executed in a cross-compiliation context on an x64 system. But
until now i couldn't figure out, if i can ignore these warnings or if
they might cause a problem.

-- 
Best regards

Christoph

I did some further digging into this issue.
it warns if the class type in the elf header is
32bit.

Christoph Buck <dev@icepic.de> writes:

> Hi!
>
> Currently i am trying to create an guix image which will boot on
> embedded imx6 arm32 board. Following the guix manual, i was able to
> create such an image. This involved adding a custom uboot version and a
> kernel with custom definition file. If flashed on an sdcard, the uboot
> runs and the kernel boots. However, early on boot (presumably on
> executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown and
> a guix recovery repl is opened
>
>> Use 'gnu.repl' for an initrd REPL.
>
>> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
>> Throw to key `record-abi-mismatch-error' with args `(abi-check "~a: record ABI mismatch; recompilation needed" (#<record-type <file-system>>) ())'.
>
>> Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
>> GNU Guile 3.0.9
>> Copyright (C) 1995-2023 Free Software Foundation, Inc.
>
>> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
>> This program is free software, and you are welcome to redistribute it
>> under certain conditions; type `,show c' for details.
>
> Unfortunatly i have absolutely no clue what the problem could be. Could
> it be that the image was compiled with a differnt guile version than
> executet on the image? Could this explain the abi mismatch in the
> `file-system` record?
>
> Googling for the error i found the following post on this mailing list:
>
>> https://lists.gnu.org/archive/html/help-guix/2023-02/msg00147.html
>
> Seems like Maxim Cournoyer had the same problem with a board with the
> same socc (imx6). Unfortunatly no followup. (I mailed him in private in
> case he come up with a solution. If so, i will document it here, so that
> the next unlucky soul running into this error can find the solution).
>
> I cross-compile the image on x64 with 
>
>> guix build -f custom-board.scm --target=arm-linux-gnueabihf -v3 -c2 -M2 -K --no-grafts
>
> where `custom-board.scm` is my image definition (i can share it if
> helpfull). Option `--no-grafts` is needed due to
>
>> https://issues.guix.gnu.org/66866
>
> For tips on how to debug this issue further i would be very
> grateful. Feels like i am very close to a bootable image.

-- 
Best regards

Christoph