ABI mismatch on boot on arm32 system

ABI mismatch on boot on arm32 system

[Christoph Buck]

Hi!

Currently i am trying to create an guix image which will boot on
embedded imx6 arm32 board. Following the guix manual, i was able to
create such an image. This involved adding a custom uboot version and a
kernel with custom definition file. If flashed on an sdcard, the uboot
runs and the kernel boots. However, early on boot (presumably on
executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown and
a guix recovery repl is opened

> Use 'gnu.repl' for an initrd REPL.

> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> Throw to key `record-abi-mismatch-error' with args `(abi-check "~a: record ABI mismatch; recompilation needed" (#<record-type <file-system>>) ())'.

> Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
> GNU Guile 3.0.9
> Copyright (C) 1995-2023 Free Software Foundation, Inc.

> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.

Unfortunatly i have absolutely no clue what the problem could be. Could
it be that the image was compiled with a differnt guile version than
executet on the image? Could this explain the abi mismatch in the
`file-system` record?

Googling for the error i found the following post on this mailing list:

> https://lists.gnu.org/archive/html/help-guix/2023-02/msg00147.html

Seems like Maxim Cournoyer had the same problem with a board with the
same socc (imx6). Unfortunatly no followup. (I mailed him in private in
case he come up with a solution. If so, i will document it here, so that
the next unlucky soul running into this error can find the solution).

I cross-compile the image on x64 with 

> guix build -f custom-board.scm --target=arm-linux-gnueabihf -v3 -c2 -M2 -K --no-grafts

where `custom-board.scm` is my image definition (i can share it if
helpfull). Option `--no-grafts` is needed due to

> https://issues.guix.gnu.org/66866

For tips on how to debug this issue further i would be very
grateful. Feels like i am very close to a bootable image.

-- 
Best regards

Christoph

Re: ABI mismatch on boot on arm32 system

[Richard Sent]

Hi Christoph,

> Currently i am trying to create an guix image which will boot on
> embedded imx6 arm32 board. Following the guix manual, i was able to
> create such an image. This involved adding a custom uboot version and a
> kernel with custom definition file. If flashed on an sdcard, the uboot
> runs and the kernel boots. However, early on boot (presumably on
> executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown and
> a guix recovery repl is opened
>
>> Use 'gnu.repl' for an initrd REPL.
>
>> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
>> Throw to key `record-abi-mismatch-error' with args `(abi-check "~a: record ABI mismatch; recompilation needed" (#<record-type <file-system>>) ())'.

Your issue sounds very similar to the one described in
https://issues.guix.gnu.org/61173#4.

The TL;DR (although I encourage you to read it!) is that you need the
CONFIG_BINFMT_MISC Linux kernel compilation option set, but when you use
a linux-libre-*-generic kernel that option is NOT set. If you're using
the qemu-binfmt-service, you'll fail to boot and the error you posted
will occur before you're dropped into a REPL.

Unfortunately services do not currently have any mechanism to require or
check kernel config options.

Not knowing your operating-system declaration I can't tell for sure if
that is what's going on, but I suspect what I described or something
similar is the case.

If you confirm this is in fact the problem, feel free to leave a comment
on the issue!

Best of luck.

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Re: ABI mismatch on boot on arm32 system

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On Wed, 16 Oct 2024 12:11:30 +0200
Christoph Buck <dev@icepic.de> wrote:

> Hi!
Hi,

> Currently i am trying to create an guix image which will boot on
> embedded imx6 arm32 board. Following the guix manual, i was able to
> create such an image. This involved adding a custom uboot version and
> a kernel with custom definition file. If flashed on an sdcard, the
> uboot runs and the kernel boots. However, early on boot (presumably on
> executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown
> and a guix recovery repl is opened
> 
> > Use 'gnu.repl' for an initrd REPL.
> 
> > ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> > Throw to key `record-abi-mismatch-error' with args `(abi-check "~a:
> > record ABI mismatch; recompilation needed" (#<record-type
> > <file-system>>) ())'.
There is also the option to try to bisect the issues (there might be
more than one).

We now have a u-boot-qemu-arm package so you could for instance start
with arm64 (with u-boot-qemu-arm64 and a system definition that you
create or reuse+modify) and manage to boot a system with qemu.

Then once you managed to boot an arm64 system, you could try to
reproduce it for 32bit arm with an older guix revision (and possibly a
recent u-boot-qemu-arm that doesn't change) and then start bisecting.

I tried to do that a long time ago but I don't have fast computers and
so at some point I gave up and I never found the time to get back to it.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: ABI mismatch on boot on arm32 system

[Christoph Buck]

Richard Sent <richard@freakingpenguin.com> writes:

> Hi Christoph,

Hi Richard!
>
> The TL;DR (although I encourage you to read it!) is that you need the
> CONFIG_BINFMT_MISC Linux kernel compilation option set, but when you use
> a linux-libre-*-generic kernel that option is NOT set. If you're using
> the qemu-binfmt-service, you'll fail to boot and the error you posted
> will occur before you're dropped into a REPL.

I use a custom kernel modified straight from kernel.org and indeed i didn't
enable the `CONFIG_BINFMT_MISC` setting. However, unfortunately enabling
this option **does not** solve my problem.


-- 
Best regards

Christoph

Re: ABI mismatch on boot on arm32 system

[Christoph Buck]

Hi!

I played around a little bit more and i can indeed now successfully
boot. Instead of using cross-compilation (cli option
`--target=arm-linux-gnueabihf`) i created a build using qemu emulation
(cli option `--system=armhf-linux`). This takes ages to build, but the
resulting images is bootable without abi error. Unfortunatly this is not
a real fix because it is too slow to be a practical workaround (at least
for me).

I digged a little deeper and this is what i found out so far. In case i
am running off in a totally wrong direction, someone with more clue than
me should please stop me ;)

I think something goes wrong during crosscompilation of the guile
modules in package `module-import-compiled`. The abi error is thrown
early on boot in the `initrd.cpio.gz` ramdisk. I extracted and
decompressed the ramdisk from both builds (crosscompilation and qemu)
which contain the `module-import-compiled` guile modules. I would expect
that the *.go files from the `module-import-compiled` package of both
ramdisks are binary identical but they have different md5sums. Lets take
for example `file-systems.go`, which cause the abi error.

--8<---------------cut here---------------start------------->8---
local@host:crosscompilation-initrd/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system$
md5sum file-systems.go
7839e9c7a0c7c6c8d9ea45566ab9f61e  file-systems.go
--8<---------------cut here---------------end--------------->8---
vs
--8<---------------cut here---------------start------------->8---

local@host:qemu-initrd/gnu/store/hvgj80xqf70mvx460pnvwmi87kqqn2bj-module-import-compiled/gnu/system$
md5sum file-systems.go
a43a7e939ae9d0cc1ce30726cb51d6d4  file-systems.go
--8<---------------cut here---------------end--------------->8---

Additional it looks like different symbols are exported depending if
cross-compilation or qemu was used. This is at least what `readelf -s file-system.go`
reports. I naively thought these files should be identical.

Additional i saw these strange errors in the build log during
crosscompilation

--8<---------------cut here---------------start------------->8---
;;; WARNING: loading compiled file
/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/build/file-systems.go
failed:
;;; In procedure load-thunk-from-memory: ELF file does not have native word size
;;; WARNING: loading compiled file
/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system/uuid.go
failed:
;;; In procedure load-thunk-from-memory: ELF file does not have native word size
;;; WARNING: loading compiled file
/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system/file-systems.go
failed:
;;; In procedure load-thunk-from-memory: ELF file does not have native word size
--8<---------------cut here---------------end--------------->8---

This also looks suspicious. These stem from the `check_elf_header`
function in guile. Guile warns that the class type in the elf header is
32bits if executed in a cross-compiliation context on an x64 system. But
until now i couldn't figure out, if i can ignore these warnings or if
they might cause a problem.

-- 
Best regards

Christoph

I did some further digging into this issue.
it warns if the class type in the elf header is
32bit.

Christoph Buck <dev@icepic.de> writes:

> Hi!
>
> Currently i am trying to create an guix image which will boot on
> embedded imx6 arm32 board. Following the guix manual, i was able to
> create such an image. This involved adding a custom uboot version and a
> kernel with custom definition file. If flashed on an sdcard, the uboot
> runs and the kernel boots. However, early on boot (presumably on
> executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown and
> a guix recovery repl is opened
>
>> Use 'gnu.repl' for an initrd REPL.
>
>> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
>> Throw to key `record-abi-mismatch-error' with args `(abi-check "~a: record ABI mismatch; recompilation needed" (#<record-type <file-system>>) ())'.
>
>> Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
>> GNU Guile 3.0.9
>> Copyright (C) 1995-2023 Free Software Foundation, Inc.
>
>> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
>> This program is free software, and you are welcome to redistribute it
>> under certain conditions; type `,show c' for details.
>
> Unfortunatly i have absolutely no clue what the problem could be. Could
> it be that the image was compiled with a differnt guile version than
> executet on the image? Could this explain the abi mismatch in the
> `file-system` record?
>
> Googling for the error i found the following post on this mailing list:
>
>> https://lists.gnu.org/archive/html/help-guix/2023-02/msg00147.html
>
> Seems like Maxim Cournoyer had the same problem with a board with the
> same socc (imx6). Unfortunatly no followup. (I mailed him in private in
> case he come up with a solution. If so, i will document it here, so that
> the next unlucky soul running into this error can find the solution).
>
> I cross-compile the image on x64 with 
>
>> guix build -f custom-board.scm --target=arm-linux-gnueabihf -v3 -c2 -M2 -K --no-grafts
>
> where `custom-board.scm` is my image definition (i can share it if
> helpfull). Option `--no-grafts` is needed due to
>
>> https://issues.guix.gnu.org/66866
>
> For tips on how to debug this issue further i would be very
> grateful. Feels like i am very close to a bootable image.

-- 
Best regards

Christoph

Re: ABI mismatch on boot on arm32 system

[Zack Weinberg]

On Sun, Oct 20, 2024, at 11:23 AM, Christoph Buck wrote:
> I think something goes wrong during crosscompilation of the guile
> modules in package `module-import-compiled`. The abi error is thrown
> early on boot in the `initrd.cpio.gz` ramdisk. I extracted and
> decompressed the ramdisk from both builds (crosscompilation and qemu)
> which contain the `module-import-compiled` guile modules. I would expect
> that the *.go files from the `module-import-compiled` package of both
> ramdisks are binary identical but they have different md5sums. Lets take
> for example `file-systems.go`, which cause the abi error.
[...]

Can you show us the *complete and unedited* output of `readelf -hlSd file-systems.go` from both the working and the broken ramdisk, please?

zw

Re: ABI mismatch on boot on arm32 system

[Christoph Buck]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

Hi Zack!

> Can you show us the *complete and unedited* output of `readelf -hlSd
> file-systems.go` from both the working and the broken ramdisk, please?
>

Sure. See the attachments of this mail.

But i just saw that i made a mistake and compaired the module
`qemu/gnu/build/file-systems.go` to
`cross/gnu/system/file-systems.go`.

The md5sum between `qemu/gnu/system/file-systems.go` and
`cross/gnu/system/file-systems.go` is still different, but the exported
symbols are the same (see attachemt). The only difference i now can see
is in the `Start of section headers` (426376 vs 426352) and different
addresses in the subsequent output of `readelf`. Are theses expected to
be deterministic/equal?

Sorry for the confusion.

> zw

Greetings Christoph 

-- 
Best regards

Christoph


[-- Attachment #2: qemu.readelf --]
[-- Type: application/octet-stream, Size: 3734 bytes --]

ELF Header:
  Magic:   7f 45 4c 46 01 01 01 ff 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            <unknown: ff>
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           None
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          52 (bytes into file)
  Start of section headers:          426352 (bytes into file)
  Flags:                             0x0
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         3
  Size of section headers:           40 (bytes)
  Number of section headers:         20
  Section header string table index: 17

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .guile.procprops  PROGBITS        00000000 06c790 000020 00      0   0  8
  [ 2] .rodata           PROGBITS        0003fe38 03fe38 0074ac 00   A  0   0  8
  [ 3] .data             PROGBITS        00050000 050000 018170 00  WA  0   0  8
  [ 4] .rtl-text         PROGBITS        00000098 000098 03fda0 00   A  0   0  8
  [ 5] .dynamic          DYNAMIC         000472e8 0472e8 000030 00   A  0   0  8
  [ 6] .strtab           STRTAB          00000000 070b98 000666 00      0   0  8
  [ 7] .symtab           SYMTAB          00000000 06f118 001a80 10      6   0  8
  [ 8] .guile.ariti[...] STRTAB          00000000 071200 00085a 00      0   0  8
  [ 9] .guile.arities    PROGBITS        00000000 068490 004253 00      8   0  8
  [10] .guile.docst[...] STRTAB          00000000 071a60 0008ac 00      0   0  8
  [11] .guile.docstrs    PROGBITS        00000000 06c6e8 0000a8 00     10   0  8
  [12] .debug_info       PROGBITS        00000000 06c7b0 000b6f 00      0   0  8
  [13] .debug_abbrev     PROGBITS        00000000 06d320 000041 00      0   0  8
  [14] .debug_str        PROGBITS        00000000 06d368 000672 00      0   0  8
  [15] .debug_loc        PROGBITS        00000000 06d9e0 000000 00      0   0  8
  [16] .debug_line       PROGBITS        00000000 06d9e0 001733 00      0   0  8
  [17] .shstrtab         STRTAB          00000000 072310 0000d3 00      0   0  8
  [18]                   PROGBITS        00000000 000000 000094 00   A  0   0  8
  [19]                   NULL            00000000 068170 000000 00      0   0  0
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x000000 0x00000000 0x00000000 0x47318 0x47318 R   0x10000
  LOAD           0x050000 0x00050000 0x00050000 0x18170 0x18170 RW  0x10000
  DYNAMIC        0x0472e8 0x000472e8 0x000472e8 0x00030 0x00030 R   0x8

 Section to Segment mapping:
  Segment Sections...
   00     .rodata .rtl-text .dynamic  
   01     .data 
   02     .dynamic 

Dynamic section at offset 0x472e8 contains 6 entries:
  Tag        Type                         Name/Value
 0x37146003 (<unknown>: 37146003)        0x3000006
 0x37146002 (<unknown>: 37146002)        0x98
 0x37146000 (<unknown>: 37146000)        0x50000
 0x37146001 (<unknown>: 37146001)        0x18170
 0x0000000c (INIT)                       0x1b918
 0x00000000 (NULL)                       0x0

[-- Attachment #3: cross.readelf --]
[-- Type: application/octet-stream, Size: 3735 bytes --]

ELF Header:
  Magic:   7f 45 4c 46 01 01 01 ff 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            <unknown: ff>
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           None
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          52 (bytes into file)
  Start of section headers:          426376 (bytes into file)
  Flags:                             0x0
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         3
  Size of section headers:           40 (bytes)
  Number of section headers:         20
  Section header string table index: 17

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .guile.procprops  PROGBITS        00000000 06c7a8 000020 00      0   0  8
  [ 2] .rodata           PROGBITS        0003fe60 03fe60 008b64 00   A  0   0  8
  [ 3] .data             PROGBITS        00050000 050000 018188 00  WA  0   0  8
  [ 4] .rtl-text         PROGBITS        00000098 000098 03fdc4 00   A  0   0  8
  [ 5] .dynamic          DYNAMIC         000489c8 0489c8 000030 00   A  0   0  8
  [ 6] .strtab           STRTAB          00000000 070ba8 000666 00      0   0  8
  [ 7] .symtab           SYMTAB          00000000 06f128 001a80 10      6   0  8
  [ 8] .guile.ariti[...] STRTAB          00000000 071210 000b62 00      0   0  8
  [ 9] .guile.arities    PROGBITS        00000000 0684a8 004255 00      8   0  8
  [10] .guile.docst[...] STRTAB          00000000 071d78 0008ac 00      0   0  8
  [11] .guile.docstrs    PROGBITS        00000000 06c700 0000a8 00     10   0  8
  [12] .debug_info       PROGBITS        00000000 06c7c8 000b67 00      0   0  8
  [13] .debug_abbrev     PROGBITS        00000000 06d330 000041 00      0   0  8
  [14] .debug_str        PROGBITS        00000000 06d378 000672 00      0   0  8
  [15] .debug_loc        PROGBITS        00000000 06d9f0 000000 00      0   0  8
  [16] .debug_line       PROGBITS        00000000 06d9f0 001733 00      0   0  8
  [17] .shstrtab         STRTAB          00000000 072628 0000d3 00      0   0  8
  [18]                   PROGBITS        00000000 000000 000094 00   A  0   0  8
  [19]                   NULL            00000000 068188 000000 00      0   0  0
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclupde),
  p (processor specific)

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x000000 0x00000000 0x00000000 0x489f8 0x489f8 R   0x10000
  LOAD           0x050000 0x00050000 0x00050000 0x18188 0x18188 RW  0x10000
  DYNAMIC        0x0489c8 0x000489c8 0x000489c8 0x00030 0x00030 R   0x8

 Section to Segment mapping:
  Segment Sections...
   00     .rodata .rtl-text .dynamic  
   01     .data 
   02     .dynamic 

Dynamic section at offset 0x489c8 contains 6 entries:
  Tag        Type                         Name/Value
 0x37146003 (<unknown>: 37146003)        0x3000006
 0x37146002 (<unknown>: 37146002)        0x98
 0x37146000 (<unknown>: 37146000)        0x50000
 0x37146001 (<unknown>: 37146001)        0x18188
 0x0000000c (INIT)                       0x1b918
 0x00000000 (NULL)                       0x0