Hi, I found the underlying cause of the problem with qemu transparent emulation: * qemu transparent emulator has 64 bit registers * the thing it's emulating has 32 bit registers * The glibc in the distro that is running in the emulator is using getdents64 (on 32 bits!) and then (rightfully) checking whether d_off and the inode number fit into their own (32 bits/entry) struct, which they don't (the thing they get from the kernel is 64 bits/entry). See also https://lore.kernel.org/lkml/20181229015453.GA6310@bombadil.infradead.org/T/ for an analysis. See also https://sourceware.org/bugzilla/show_bug.cgi?id=23960 for a discussion. Least-shitty workaround: Use a 32-bit qemu (yes, a qemu compiled on 32 bit) on a 64 bit machine for transparent emulation of ANOTHER 32-bit machine. That way, the kernel can know that there's a 32 bit user lurking somewhere up the call chain that is calling getdents64 and is not actually able to process the result. "The truth? It can't handle the truth." The right fix: One could also tell all the packages in the emulated system to use the large file size API (-D_FILE_OFFSET_BITS=64 and co). In this case cmake is affected--but it could be any number of things. I think that that is the only good fix (we could also add a compile-time check whether has been included without anyone specifying -D_FILE_OFFSET_BITS=64--that would make finding these problems a LOT easier; if possible, emit that compile-time error only if readdir is actually called anywhere). For the workaround, we could adapt Guix system's gnu/services/virtualization.scm so it uses a qemu built for the 32-bit variant of the host architecture if it's emulating a 32 bit system. So qemu could be compiled for i686 if the host is x86_64 and the emulated system is armhf, qemu could be compiled for armhf if the host is aarch64 and the emulated system is armhf and so on. That also means that if a host system is 64 bits and DOES NOT HAVE a 32 bit pendant target, then the emulation of a 32 bit system is going to be imperfect. One way to fix that would be to fix the kernel to accept an argument on the getdents64 syscall (and similar ones) that tells it whether the user wants to have 64 bit offsets or 32 bit offsets[1]. Right now, from user space, that is only possible via process personality flags. And those would switch the entire qemu executable over to 32 bits, which we don't want (qemu itself has to do stuff using kernel syscalls, so it needs to be capable of 64 bits if it itself is 64 bits). And I think that even that case is not being handled in the kernel correctly. So this fix cannot be done. @Ludo: Anyway, I have a question on how to replicate what "guix build --target=i686..." does, in the guix service definition. How can I make it use package-cross-derivation instead of package-derivation ? See attachment. With the attachment and the service definition (service qemu-binfmt-service-type (qemu-binfmt-configuration (platforms (lookup-qemu-platforms "arm")) (guix-support? #t))) in order to emulate ARM on x86_64 I eventually get: >Building qemu-minimal for i686 (That's what I want!) >[...] >starting phase `configure' > >ERROR: pkg-config binary 'i686-unknown-linux-gnu-pkg-config' not found pkg-config has not been cross-compiled... That's because it's not using package-cross-derivation, I guess. >command "./configure" "--cc=/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/bin/gcc" "--host-cc=/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/bin/gcc" "--disable-debug-info" "--enable-virtfs" "--prefix=/gnu/store/80ljf47lrh8arrzjmkrrqxghc0k67b3s-qemu-minimal-5.1.0" "--sysconfdir=/etc" "--cross-prefix=i686-unknown-linux-gnu-" "--target-list=i386-softmmu,x86_64-softmmu" failed with status 1 I think that "--cc" should use ,(cc-for-target) at all times. Better this: diff --git a/gnu/packages/virtualization.scm b/gnu/packages/virtualization.scm index 53e9dde125..bf712afd4a 100644 --- a/gnu/packages/virtualization.scm +++ b/gnu/packages/virtualization.scm @@ -227,7 +227,7 @@ (setenv "LDFLAGS" "-lrt") (apply invoke `("./configure" - ,(string-append "--cc=" (which "gcc")) + ,(string-append "--cc=" ,(cc-for-target)) ;; Some architectures insist on using HOST_CC ,(string-append "--host-cc=" (which "gcc")) "--disable-debug-info" ; save build space [1] A way for userspace to tell the kernel that is to use getdents instad of getdents64 on 32 bits, like it used to. But they don't want to do that anymore. Another way would be for qemu to translate a syscall getdents64 from the guest to getdents on the host if the machine they are emulating is 32 bits. But that would mean that getdents on a 64 bit host would still have to act like it's 32 bits and translate the d_off accordingly. That's not guaranteed[2]. Or even using the old readdir syscall. [2] Linux-5.8.8: [... CONFIG_COMPAT ...] SYSCALL_DEFINE3(getdents, unsigned int, fd, struct linux_dirent __user *, dirent, unsigned int, count) { struct fd f; struct getdents_callback buf = { .ctx.actor = filldir, .count = count, .current_dir = dirent }; int error; f = fdget_pos(fd); if (!f.file) return -EBADF; error = iterate_dir(f.file, &buf.ctx); if (error >= 0) error = buf.error; if (buf.prev_reclen) { struct linux_dirent __user * lastdirent; lastdirent = (void __user *)buf.current_dir - buf.prev_reclen; if (put_user(buf.ctx.pos, &lastdirent->d_off)) error = -EFAULT; else error = count - buf.count; } fdput_pos(f); return error; } It's not guaranteed.