* ABI mismatch on boot on arm32 system @ 2024-10-16 10:11 Christoph Buck 2024-10-16 20:05 ` Richard Sent ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Christoph Buck @ 2024-10-16 10:11 UTC (permalink / raw) To: help-guix Hi! Currently i am trying to create an guix image which will boot on embedded imx6 arm32 board. Following the guix manual, i was able to create such an image. This involved adding a custom uboot version and a kernel with custom definition file. If flashed on an sdcard, the uboot runs and the kernel boots. However, early on boot (presumably on executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown and a guix recovery repl is opened > Use 'gnu.repl' for an initrd REPL. > ice-9/boot-9.scm:1685:16: In procedure raise-exception: > Throw to key `record-abi-mismatch-error' with args `(abi-check "~a: record ABI mismatch; recompilation needed" (#<record-type <file-system>>) ())'. > Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. > GNU Guile 3.0.9 > Copyright (C) 1995-2023 Free Software Foundation, Inc. > Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. > This program is free software, and you are welcome to redistribute it > under certain conditions; type `,show c' for details. Unfortunatly i have absolutely no clue what the problem could be. Could it be that the image was compiled with a differnt guile version than executet on the image? Could this explain the abi mismatch in the `file-system` record? Googling for the error i found the following post on this mailing list: > https://lists.gnu.org/archive/html/help-guix/2023-02/msg00147.html Seems like Maxim Cournoyer had the same problem with a board with the same socc (imx6). Unfortunatly no followup. (I mailed him in private in case he come up with a solution. If so, i will document it here, so that the next unlucky soul running into this error can find the solution). I cross-compile the image on x64 with > guix build -f custom-board.scm --target=arm-linux-gnueabihf -v3 -c2 -M2 -K --no-grafts where `custom-board.scm` is my image definition (i can share it if helpfull). Option `--no-grafts` is needed due to > https://issues.guix.gnu.org/66866 For tips on how to debug this issue further i would be very grateful. Feels like i am very close to a bootable image. -- Best regards Christoph ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: ABI mismatch on boot on arm32 system 2024-10-16 10:11 ABI mismatch on boot on arm32 system Christoph Buck @ 2024-10-16 20:05 ` Richard Sent 2024-10-20 15:15 ` Christoph Buck 2024-10-18 20:58 ` Denis 'GNUtoo' Carikli 2024-10-20 15:23 ` Christoph Buck 2 siblings, 1 reply; 7+ messages in thread From: Richard Sent @ 2024-10-16 20:05 UTC (permalink / raw) To: Christoph Buck; +Cc: help-guix Hi Christoph, > Currently i am trying to create an guix image which will boot on > embedded imx6 arm32 board. Following the guix manual, i was able to > create such an image. This involved adding a custom uboot version and a > kernel with custom definition file. If flashed on an sdcard, the uboot > runs and the kernel boots. However, early on boot (presumably on > executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown and > a guix recovery repl is opened > >> Use 'gnu.repl' for an initrd REPL. > >> ice-9/boot-9.scm:1685:16: In procedure raise-exception: >> Throw to key `record-abi-mismatch-error' with args `(abi-check "~a: record ABI mismatch; recompilation needed" (#<record-type <file-system>>) ())'. Your issue sounds very similar to the one described in https://issues.guix.gnu.org/61173#4. The TL;DR (although I encourage you to read it!) is that you need the CONFIG_BINFMT_MISC Linux kernel compilation option set, but when you use a linux-libre-*-generic kernel that option is NOT set. If you're using the qemu-binfmt-service, you'll fail to boot and the error you posted will occur before you're dropped into a REPL. Unfortunately services do not currently have any mechanism to require or check kernel config options. Not knowing your operating-system declaration I can't tell for sure if that is what's going on, but I suspect what I described or something similar is the case. If you confirm this is in fact the problem, feel free to leave a comment on the issue! Best of luck. -- Take it easy, Richard Sent Making my computer weirder one commit at a time. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: ABI mismatch on boot on arm32 system 2024-10-16 20:05 ` Richard Sent @ 2024-10-20 15:15 ` Christoph Buck 0 siblings, 0 replies; 7+ messages in thread From: Christoph Buck @ 2024-10-20 15:15 UTC (permalink / raw) To: Richard Sent; +Cc: help-guix Richard Sent <richard@freakingpenguin.com> writes: > Hi Christoph, Hi Richard! > > The TL;DR (although I encourage you to read it!) is that you need the > CONFIG_BINFMT_MISC Linux kernel compilation option set, but when you use > a linux-libre-*-generic kernel that option is NOT set. If you're using > the qemu-binfmt-service, you'll fail to boot and the error you posted > will occur before you're dropped into a REPL. I use a custom kernel modified straight from kernel.org and indeed i didn't enable the `CONFIG_BINFMT_MISC` setting. However, unfortunately enabling this option **does not** solve my problem. -- Best regards Christoph ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: ABI mismatch on boot on arm32 system 2024-10-16 10:11 ABI mismatch on boot on arm32 system Christoph Buck 2024-10-16 20:05 ` Richard Sent @ 2024-10-18 20:58 ` Denis 'GNUtoo' Carikli 2024-10-20 15:23 ` Christoph Buck 2 siblings, 0 replies; 7+ messages in thread From: Denis 'GNUtoo' Carikli @ 2024-10-18 20:58 UTC (permalink / raw) To: Christoph Buck; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 1489 bytes --] On Wed, 16 Oct 2024 12:11:30 +0200 Christoph Buck <dev@icepic.de> wrote: > Hi! Hi, > Currently i am trying to create an guix image which will boot on > embedded imx6 arm32 board. Following the guix manual, i was able to > create such an image. This involved adding a custom uboot version and > a kernel with custom definition file. If flashed on an sdcard, the > uboot runs and the kernel boots. However, early on boot (presumably on > executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown > and a guix recovery repl is opened > > > Use 'gnu.repl' for an initrd REPL. > > > ice-9/boot-9.scm:1685:16: In procedure raise-exception: > > Throw to key `record-abi-mismatch-error' with args `(abi-check "~a: > > record ABI mismatch; recompilation needed" (#<record-type > > <file-system>>) ())'. There is also the option to try to bisect the issues (there might be more than one). We now have a u-boot-qemu-arm package so you could for instance start with arm64 (with u-boot-qemu-arm64 and a system definition that you create or reuse+modify) and manage to boot a system with qemu. Then once you managed to boot an arm64 system, you could try to reproduce it for 32bit arm with an older guix revision (and possibly a recent u-boot-qemu-arm that doesn't change) and then start bisecting. I tried to do that a long time ago but I don't have fast computers and so at some point I gave up and I never found the time to get back to it. Denis. [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: ABI mismatch on boot on arm32 system 2024-10-16 10:11 ABI mismatch on boot on arm32 system Christoph Buck 2024-10-16 20:05 ` Richard Sent 2024-10-18 20:58 ` Denis 'GNUtoo' Carikli @ 2024-10-20 15:23 ` Christoph Buck 2024-10-20 15:39 ` Zack Weinberg 2 siblings, 1 reply; 7+ messages in thread From: Christoph Buck @ 2024-10-20 15:23 UTC (permalink / raw) To: help-guix Hi! I played around a little bit more and i can indeed now successfully boot. Instead of using cross-compilation (cli option `--target=arm-linux-gnueabihf`) i created a build using qemu emulation (cli option `--system=armhf-linux`). This takes ages to build, but the resulting images is bootable without abi error. Unfortunatly this is not a real fix because it is too slow to be a practical workaround (at least for me). I digged a little deeper and this is what i found out so far. In case i am running off in a totally wrong direction, someone with more clue than me should please stop me ;) I think something goes wrong during crosscompilation of the guile modules in package `module-import-compiled`. The abi error is thrown early on boot in the `initrd.cpio.gz` ramdisk. I extracted and decompressed the ramdisk from both builds (crosscompilation and qemu) which contain the `module-import-compiled` guile modules. I would expect that the *.go files from the `module-import-compiled` package of both ramdisks are binary identical but they have different md5sums. Lets take for example `file-systems.go`, which cause the abi error. --8<---------------cut here---------------start------------->8--- local@host:crosscompilation-initrd/gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system$ md5sum file-systems.go 7839e9c7a0c7c6c8d9ea45566ab9f61e file-systems.go --8<---------------cut here---------------end--------------->8--- vs --8<---------------cut here---------------start------------->8--- local@host:qemu-initrd/gnu/store/hvgj80xqf70mvx460pnvwmi87kqqn2bj-module-import-compiled/gnu/system$ md5sum file-systems.go a43a7e939ae9d0cc1ce30726cb51d6d4 file-systems.go --8<---------------cut here---------------end--------------->8--- Additional it looks like different symbols are exported depending if cross-compilation or qemu was used. This is at least what `readelf -s file-system.go` reports. I naively thought these files should be identical. Additional i saw these strange errors in the build log during crosscompilation --8<---------------cut here---------------start------------->8--- ;;; WARNING: loading compiled file /gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/build/file-systems.go failed: ;;; In procedure load-thunk-from-memory: ELF file does not have native word size ;;; WARNING: loading compiled file /gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system/uuid.go failed: ;;; In procedure load-thunk-from-memory: ELF file does not have native word size ;;; WARNING: loading compiled file /gnu/store/5ffy1h3fgikzdhfz4nkchxnibbri4ain-module-import-compiled/gnu/system/file-systems.go failed: ;;; In procedure load-thunk-from-memory: ELF file does not have native word size --8<---------------cut here---------------end--------------->8--- This also looks suspicious. These stem from the `check_elf_header` function in guile. Guile warns that the class type in the elf header is 32bits if executed in a cross-compiliation context on an x64 system. But until now i couldn't figure out, if i can ignore these warnings or if they might cause a problem. -- Best regards Christoph I did some further digging into this issue. it warns if the class type in the elf header is 32bit. Christoph Buck <dev@icepic.de> writes: > Hi! > > Currently i am trying to create an guix image which will boot on > embedded imx6 arm32 board. Following the guix manual, i was able to > create such an image. This involved adding a custom uboot version and a > kernel with custom definition file. If flashed on an sdcard, the uboot > runs and the kernel boots. However, early on boot (presumably on > executing initrd.cpio.gz), an `record-abi-mismatch-error` is thrown and > a guix recovery repl is opened > >> Use 'gnu.repl' for an initrd REPL. > >> ice-9/boot-9.scm:1685:16: In procedure raise-exception: >> Throw to key `record-abi-mismatch-error' with args `(abi-check "~a: record ABI mismatch; recompilation needed" (#<record-type <file-system>>) ())'. > >> Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. >> GNU Guile 3.0.9 >> Copyright (C) 1995-2023 Free Software Foundation, Inc. > >> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. >> This program is free software, and you are welcome to redistribute it >> under certain conditions; type `,show c' for details. > > Unfortunatly i have absolutely no clue what the problem could be. Could > it be that the image was compiled with a differnt guile version than > executet on the image? Could this explain the abi mismatch in the > `file-system` record? > > Googling for the error i found the following post on this mailing list: > >> https://lists.gnu.org/archive/html/help-guix/2023-02/msg00147.html > > Seems like Maxim Cournoyer had the same problem with a board with the > same socc (imx6). Unfortunatly no followup. (I mailed him in private in > case he come up with a solution. If so, i will document it here, so that > the next unlucky soul running into this error can find the solution). > > I cross-compile the image on x64 with > >> guix build -f custom-board.scm --target=arm-linux-gnueabihf -v3 -c2 -M2 -K --no-grafts > > where `custom-board.scm` is my image definition (i can share it if > helpfull). Option `--no-grafts` is needed due to > >> https://issues.guix.gnu.org/66866 > > For tips on how to debug this issue further i would be very > grateful. Feels like i am very close to a bootable image. -- Best regards Christoph ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: ABI mismatch on boot on arm32 system 2024-10-20 15:23 ` Christoph Buck @ 2024-10-20 15:39 ` Zack Weinberg 2024-10-20 17:24 ` Christoph Buck 0 siblings, 1 reply; 7+ messages in thread From: Zack Weinberg @ 2024-10-20 15:39 UTC (permalink / raw) To: help-guix On Sun, Oct 20, 2024, at 11:23 AM, Christoph Buck wrote: > I think something goes wrong during crosscompilation of the guile > modules in package `module-import-compiled`. The abi error is thrown > early on boot in the `initrd.cpio.gz` ramdisk. I extracted and > decompressed the ramdisk from both builds (crosscompilation and qemu) > which contain the `module-import-compiled` guile modules. I would expect > that the *.go files from the `module-import-compiled` package of both > ramdisks are binary identical but they have different md5sums. Lets take > for example `file-systems.go`, which cause the abi error. [...] Can you show us the *complete and unedited* output of `readelf -hlSd file-systems.go` from both the working and the broken ramdisk, please? zw ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: ABI mismatch on boot on arm32 system 2024-10-20 15:39 ` Zack Weinberg @ 2024-10-20 17:24 ` Christoph Buck 0 siblings, 0 replies; 7+ messages in thread From: Christoph Buck @ 2024-10-20 17:24 UTC (permalink / raw) To: Zack Weinberg; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 784 bytes --] Hi Zack! > Can you show us the *complete and unedited* output of `readelf -hlSd > file-systems.go` from both the working and the broken ramdisk, please? > Sure. See the attachments of this mail. But i just saw that i made a mistake and compaired the module `qemu/gnu/build/file-systems.go` to `cross/gnu/system/file-systems.go`. The md5sum between `qemu/gnu/system/file-systems.go` and `cross/gnu/system/file-systems.go` is still different, but the exported symbols are the same (see attachemt). The only difference i now can see is in the `Start of section headers` (426376 vs 426352) and different addresses in the subsequent output of `readelf`. Are theses expected to be deterministic/equal? Sorry for the confusion. > zw Greetings Christoph -- Best regards Christoph [-- Attachment #2: qemu.readelf --] [-- Type: application/octet-stream, Size: 3734 bytes --] ELF Header: Magic: 7f 45 4c 46 01 01 01 ff 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, little endian Version: 1 (current) OS/ABI: <unknown: ff> ABI Version: 0 Type: DYN (Shared object file) Machine: None Version: 0x1 Entry point address: 0x0 Start of program headers: 52 (bytes into file) Start of section headers: 426352 (bytes into file) Flags: 0x0 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 3 Size of section headers: 40 (bytes) Number of section headers: 20 Section header string table index: 17 Section Headers: [Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 0] NULL 00000000 000000 000000 00 0 0 0 [ 1] .guile.procprops PROGBITS 00000000 06c790 000020 00 0 0 8 [ 2] .rodata PROGBITS 0003fe38 03fe38 0074ac 00 A 0 0 8 [ 3] .data PROGBITS 00050000 050000 018170 00 WA 0 0 8 [ 4] .rtl-text PROGBITS 00000098 000098 03fda0 00 A 0 0 8 [ 5] .dynamic DYNAMIC 000472e8 0472e8 000030 00 A 0 0 8 [ 6] .strtab STRTAB 00000000 070b98 000666 00 0 0 8 [ 7] .symtab SYMTAB 00000000 06f118 001a80 10 6 0 8 [ 8] .guile.ariti[...] STRTAB 00000000 071200 00085a 00 0 0 8 [ 9] .guile.arities PROGBITS 00000000 068490 004253 00 8 0 8 [10] .guile.docst[...] STRTAB 00000000 071a60 0008ac 00 0 0 8 [11] .guile.docstrs PROGBITS 00000000 06c6e8 0000a8 00 10 0 8 [12] .debug_info PROGBITS 00000000 06c7b0 000b6f 00 0 0 8 [13] .debug_abbrev PROGBITS 00000000 06d320 000041 00 0 0 8 [14] .debug_str PROGBITS 00000000 06d368 000672 00 0 0 8 [15] .debug_loc PROGBITS 00000000 06d9e0 000000 00 0 0 8 [16] .debug_line PROGBITS 00000000 06d9e0 001733 00 0 0 8 [17] .shstrtab STRTAB 00000000 072310 0000d3 00 0 0 8 [18] PROGBITS 00000000 000000 000094 00 A 0 0 8 [19] NULL 00000000 068170 000000 00 0 0 0 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), p (processor specific) Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align LOAD 0x000000 0x00000000 0x00000000 0x47318 0x47318 R 0x10000 LOAD 0x050000 0x00050000 0x00050000 0x18170 0x18170 RW 0x10000 DYNAMIC 0x0472e8 0x000472e8 0x000472e8 0x00030 0x00030 R 0x8 Section to Segment mapping: Segment Sections... 00 .rodata .rtl-text .dynamic 01 .data 02 .dynamic Dynamic section at offset 0x472e8 contains 6 entries: Tag Type Name/Value 0x37146003 (<unknown>: 37146003) 0x3000006 0x37146002 (<unknown>: 37146002) 0x98 0x37146000 (<unknown>: 37146000) 0x50000 0x37146001 (<unknown>: 37146001) 0x18170 0x0000000c (INIT) 0x1b918 0x00000000 (NULL) 0x0 [-- Attachment #3: cross.readelf --] [-- Type: application/octet-stream, Size: 3735 bytes --] ELF Header: Magic: 7f 45 4c 46 01 01 01 ff 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, little endian Version: 1 (current) OS/ABI: <unknown: ff> ABI Version: 0 Type: DYN (Shared object file) Machine: None Version: 0x1 Entry point address: 0x0 Start of program headers: 52 (bytes into file) Start of section headers: 426376 (bytes into file) Flags: 0x0 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 3 Size of section headers: 40 (bytes) Number of section headers: 20 Section header string table index: 17 Section Headers: [Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 0] NULL 00000000 000000 000000 00 0 0 0 [ 1] .guile.procprops PROGBITS 00000000 06c7a8 000020 00 0 0 8 [ 2] .rodata PROGBITS 0003fe60 03fe60 008b64 00 A 0 0 8 [ 3] .data PROGBITS 00050000 050000 018188 00 WA 0 0 8 [ 4] .rtl-text PROGBITS 00000098 000098 03fdc4 00 A 0 0 8 [ 5] .dynamic DYNAMIC 000489c8 0489c8 000030 00 A 0 0 8 [ 6] .strtab STRTAB 00000000 070ba8 000666 00 0 0 8 [ 7] .symtab SYMTAB 00000000 06f128 001a80 10 6 0 8 [ 8] .guile.ariti[...] STRTAB 00000000 071210 000b62 00 0 0 8 [ 9] .guile.arities PROGBITS 00000000 0684a8 004255 00 8 0 8 [10] .guile.docst[...] STRTAB 00000000 071d78 0008ac 00 0 0 8 [11] .guile.docstrs PROGBITS 00000000 06c700 0000a8 00 10 0 8 [12] .debug_info PROGBITS 00000000 06c7c8 000b67 00 0 0 8 [13] .debug_abbrev PROGBITS 00000000 06d330 000041 00 0 0 8 [14] .debug_str PROGBITS 00000000 06d378 000672 00 0 0 8 [15] .debug_loc PROGBITS 00000000 06d9f0 000000 00 0 0 8 [16] .debug_line PROGBITS 00000000 06d9f0 001733 00 0 0 8 [17] .shstrtab STRTAB 00000000 072628 0000d3 00 0 0 8 [18] PROGBITS 00000000 000000 000094 00 A 0 0 8 [19] NULL 00000000 068188 000000 00 0 0 0 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclupde), p (processor specific) Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align LOAD 0x000000 0x00000000 0x00000000 0x489f8 0x489f8 R 0x10000 LOAD 0x050000 0x00050000 0x00050000 0x18188 0x18188 RW 0x10000 DYNAMIC 0x0489c8 0x000489c8 0x000489c8 0x00030 0x00030 R 0x8 Section to Segment mapping: Segment Sections... 00 .rodata .rtl-text .dynamic 01 .data 02 .dynamic Dynamic section at offset 0x489c8 contains 6 entries: Tag Type Name/Value 0x37146003 (<unknown>: 37146003) 0x3000006 0x37146002 (<unknown>: 37146002) 0x98 0x37146000 (<unknown>: 37146000) 0x50000 0x37146001 (<unknown>: 37146001) 0x18188 0x0000000c (INIT) 0x1b918 0x00000000 (NULL) 0x0 ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-10-20 17:24 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-10-16 10:11 ABI mismatch on boot on arm32 system Christoph Buck 2024-10-16 20:05 ` Richard Sent 2024-10-20 15:15 ` Christoph Buck 2024-10-18 20:58 ` Denis 'GNUtoo' Carikli 2024-10-20 15:23 ` Christoph Buck 2024-10-20 15:39 ` Zack Weinberg 2024-10-20 17:24 ` Christoph Buck
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).