Qemu binfmt on foreign distro

Qemu binfmt on foreign distro

[Mathieu Othacehe]

Hello,

When running a guix build command with --system argument set to a
foreign architecture, guix-daemon will use linux binfmt mechanism to
emulate binary execution.

On a Guix system, the qemu-binfmt-service-type service has to be
installed, so that linux binfmt mechanism is hooked to QEMU. This
service also adds a bunch of "--chroot-directory" arguments to
guix-daemon command line, so that QEMU and all its dependencies are
accessible from the guix-daemon chroot.

On a foreign distro, one can install QEMU and setup the binfmt mechanism
(installing binfmt-support and qemu-user-static packages on
Ubuntu). However, without the proper --chroot-directory arguments passed
to guix-daemon, all binaries execution relying on binfmt will throw
ENOENT from within the guix-daemon.

So I can see two solutions here:

* Remove --system argument support for guix on foreign distributions
(except when using this argument does not require binfmt support, 32
bits build on a 64 bits machine).

* Provide some kind of script that generates the correct guix-daemon
command line, enabling binfmt support on foreign distros.

Anyhow, the current behaviour where guix build --system=... fails
misteriously because of an ENOENT error on a foreign distro is not
acceptable in my opinion.

What do you think?

Thanks,

Mathieu

Re: Qemu binfmt on foreign distro

[Ludovic Courtès]

Hi,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> On a foreign distro, one can install QEMU and setup the binfmt mechanism
> (installing binfmt-support and qemu-user-static packages on
> Ubuntu). However, without the proper --chroot-directory arguments passed
> to guix-daemon, all binaries execution relying on binfmt will throw
> ENOENT from within the guix-daemon.

Yes, so what?  :-)

Commit 7bf2a70a4ffd976d50638d3b9f2ec409763157df expects ENOEXEC in this
case, and a meaningful error message is provided when that happens.

How did you get ENOENT?  Was it because you had installed qemu/binfmt
support from the host distro, and thus /usr/bin/qemu-whatever wasn’t
found?

> So I can see two solutions here:
>
> * Remove --system argument support for guix on foreign distributions
> (except when using this argument does not require binfmt support, 32
> bits build on a 64 bits machine).
>
> * Provide some kind of script that generates the correct guix-daemon
> command line, enabling binfmt support on foreign distros.
>
> Anyhow, the current behaviour where guix build --system=... fails
> misteriously because of an ENOENT error on a foreign distro is not
> acceptable in my opinion.

How does it fail exactly, and what do you have in
/proc/sys/fs/binfmt_misc?

On Guix System, I get the expected behavior:

--8<---------------cut here---------------start------------->8---
$ sudo herd stop qemu-binfmt
Service qemu-binfmt has been stopped.
$ guix build -s aarch64-linux hello --check
La jena derivo estos konstruata:
   /gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv
building /gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv...
@ unsupported-platform /gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv aarch64-linux
while setting up the build environment: a `aarch64-linux' is required to build `/gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv', but I am a `x86_64-linux'
builder for `/gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv' failed with exit code 1
build of /gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv failed
View build log at '/var/log/guix/drvs/3j/4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv.bz2'.
guix build: error: build of `/gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv' failed
--8<---------------cut here---------------end--------------->8---

Thanks,
Ludo’.

Re: Qemu binfmt on foreign distro

[Mathieu Othacehe]

Hello Ludo,

> How did you get ENOENT?  Was it because you had installed qemu/binfmt
> support from the host distro, and thus /usr/bin/qemu-whatever wasn’t
> found?

Yes that's it.

mathieu@cervin:~$ ls /proc/sys/fs/binfmt_misc/ -al
total 0
-rw-r--r-- 1 root root 0 nov.  23 19:27 qemu-aarch64
-rw-r--r-- 1 root root 0 nov.  23 19:27 qemu-arm
...

>
> How does it fail exactly, and what do you have in
> /proc/sys/fs/binfmt_misc?
>
> On Guix System, I get the expected behavior:
>
> --8<---------------cut here---------------start------------->8---
> $ sudo herd stop qemu-binfmt
> Service qemu-binfmt has been stopped.
> $ guix build -s aarch64-linux hello --check
> La jena derivo estos konstruata:
>    /gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv
> building /gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv...
> @ unsupported-platform /gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv aarch64-linux
> while setting up the build environment: a `aarch64-linux' is required to build `/gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv', but I am a `x86_64-linux'
> builder for `/gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv' failed with exit code 1
> build of /gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv failed
> View build log at '/var/log/guix/drvs/3j/4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv.bz2'.
> guix build: error: build of `/gnu/store/3j4nj1y86szk6ibaqp4yc09m2p1xmkr5-hello-2.10.drv' failed
> --8<---------------cut here---------------end--------------->8---
>
> Thanks,
> Ludo’.

Re: Qemu binfmt on foreign distro

[Mathieu Othacehe]

Oops pressed C-c C-c by mistake, sorry :p

So here is what is installed by Ubuntu:

--8<---------------cut here---------------start------------->8---
mathieu@cervin:~/guix$ ls /proc/sys/fs/binfmt_misc/ -al
-rw-r--r-- 1 root root 0 nov.  23 19:42 qemu-aarch64
-rw-r--r-- 1 root root 0 nov.  23 19:42 qemu-alpha
--8<---------------cut here---------------end--------------->8---

and,

--8<---------------cut here---------------start------------->8---
mathieu@cervin:~/guix$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 
enabled
interpreter /usr/bin/qemu-aarch64
flags: OC
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff
--8<---------------cut here---------------end--------------->8---

Then when running:

guix build --system=armhf-linux hello

--8<---------------cut here---------------start------------->8---
The following derivation will be built:
   /gnu/store/174lzlz5ggw1gjck7qzkyw9qbi6izxql-hello-2.10.drv
building /gnu/store/174lzlz5ggw1gjck7qzkyw9qbi6izxql-hello-2.10.drv...
while setting up the build environment: executing `/gnu/store/50f18mlbj915vydqmm6nyvkh2b23vmf0-guile-2.2.6/bin/guile': No such file or directory
builder for `/gnu/store/174lzlz5ggw1gjck7qzkyw9qbi6izxql-hello-2.10.drv' failed with exit code 1
build of /gnu/store/174lzlz5ggw1gjck7qzkyw9qbi6izxql-hello-2.10.drv failed
View build log at '/var/log/guix/drvs/17/4lzlz5ggw1gjck7qzkyw9qbi6izxql-hello-2.10.drv.bz2'.
guix build: error: build of `/gnu/store/174lzlz5ggw1gjck7qzkyw9qbi6izxql-hello-2.10.drv' failed
--8<---------------cut here---------------end--------------->8---

And stracing guix daemon:

--8<---------------cut here---------------start------------->8---
[pid 21571] execve("/gnu/store/50f18mlbj915vydqmm6nyvkh2b23vmf0-guile-2.2.6/bin/guile", ["guile", "--no-auto-compile", "-L", "/gnu/store/6y2hmc19rywag4p4akfl4g3vz4cv2948-module-import", "/gnu/store/dccz535kqh527828ilsb4lyicx9xbn7p-hello-2.10-guile-builder"], 0x1fcda60 /* 12 vars */ <unfinished ...>
[pid 21557] write(4, "gmlo\0\0\0\0\250\0\0\0\0\0\0\0@ build-started /gnu/store/174lzlz5ggw1gjck7qzkyw9qbi6izxql-hello-2.10.drv - armhf-linux /var/log/guix/drvs/17//4lzlz5ggw1gjck7qzkyw9qbi6izxql-hello-2.10.drv.bz2 21571\n", 184) = 184
[pid 21557] select(20, [19], NULL, NULL, NULL <unfinished ...>
[pid 21571] <... execve resumed> )      = -1 ENOENT (No such file or directory)
--8<---------------cut here---------------end--------------->8---

However, just noticed that if I install qemu-user-static instead of
qemu-user-binfmt, everything works well, not sure why :(.

So I guess I was making noise for nothing, sorry about that.

Mathieu

Re: Qemu binfmt on foreign distro

[Mathieu Othacehe]

I ran further tests. On Ubuntu 18.04, using qemu-user-static or
qemu-user-binfmt makes no difference, they both fail.

qemu-user-static works on Ubuntu 19.04 because of the addition of the
'F' flag, see:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1815100.

Adding the 'F' flag[1] manually on Ubuntu 18.04 also fixes the issue.

I could maybe add an entry to the documentation explaining that, in
order for '--system' command to work on foreign distros, it is needed
that:

* A static version of qemu-user is used by binfmt
* The 'F' flag is enabled for all binfmt installed entries

Mathieu

[1]: https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

Re: Qemu binfmt on foreign distro

[Ludovic Courtès]

Hi,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> I ran further tests. On Ubuntu 18.04, using qemu-user-static or
> qemu-user-binfmt makes no difference, they both fail.
>
> qemu-user-static works on Ubuntu 19.04 because of the addition of the
> 'F' flag, see:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1815100.
>
> Adding the 'F' flag[1] manually on Ubuntu 18.04 also fixes the issue.

Right, so there are two things you need for emulated builds to work: (1)
the qemu-xyz program must be visible in the container, and (2) the
dynamic linker and shared libraries it refers to must also be visible.

The ‘F’ flag solves #1, and using a statically-linked QEMU solves #2.

> I could maybe add an entry to the documentation explaining that, in
> order for '--system' command to work on foreign distros, it is needed
> that:
>
> * A static version of qemu-user is used by binfmt
> * The 'F' flag is enabled for all binfmt installed entries

That has to be worded carefully: it’s only for foreign distros, and only
for architectures that the CPU doesn’t support (x86_64 can do i686, and
some AArch64 CPUs can do ARMv7).

Perhaps it could be a sub-section under “Application Setup”, linked from
the ‘--system’ documentation?

Thanks,
Ludo’.