Implications of QEMU binfmt transparent emulation for builds

Implications of QEMU binfmt transparent emulation for builds

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1781 bytes --]

Hey,

I'm starting to play with mixing native and emulated builds with the
Guix Build Coordinator again. I did do this many months ago, but at that
time, there wasn't support for targeting retries across a range of
machines, to help avoid blockages due to QEMU issues.

Anyway, something that's been on my mind regarding QEMU and builds is
how well this matches up with building natively. In particular, I'm
concerned that there are some derivations that will build on system A
with some QEMU configuration allowing binaries for system B to be run,
but won't build if that QEMU support wasn't there. I think there's also
a chance that you could have a derivation for system A that builds on a
system with QEMU support for system B, but then wouldn't build natively
on system B.

I think the first scenario is more likely, mainly because I wonder if
this happens with cross built packages. If the package attempts to run
software built for the target system, that will work if there's QEMU
support there for that target system, but not if that QEMU support is
lacking.

This is just a theory at this point, I at least don't know of any cases
of this happening, but I also haven't been looking. Any ideas?

I'd like the Guix Build Coordinator agents to be able to report the
configuration of the machine at the time builds happen, so maybe once
that's a feature, and QEMU support can be detected, then it'll be
possible to look for cases where a change in QEMU support affects the
build result.

I've been thinking about this lately because I'd like to maybe use QEMU
to help build things for aarch64-linux and armhf-linux, but I don't want
to do something that could impact x86_64-linux derivations building for
other architectures (which guix.cbaines.net does do).

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implications of QEMU binfmt transparent emulation for builds

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

On Sat, 2021-03-06 at 21:35 +0000, Christopher Baines wrote:

In general I think binary translation is very wasteful use of computing
resources. At least use something like this: 
http://csl.iis.sinica.edu.tw/hqemu/ - otherwise I think building
natively is at the end less costly.

> I'd like the Guix Build Coordinator agents to be able to report the
> configuration of the machine at the time builds happen, so maybe once
> that's a feature, and QEMU support can be detected, then it'll be
> possible to look for cases where a change in QEMU support affects the
> build result.

What about including such metadata in Nars directly? Kernel name and
version string, system/target, processor model.. or is that too
privacy-invasive..? Maybe make such opt-in.

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Implications of QEMU binfmt transparent emulation for builds

[Mark H Weaver]

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

> I'm starting to play with mixing native and emulated builds with the
> Guix Build Coordinator again. I did do this many months ago, but at that
> time, there wasn't support for targeting retries across a range of
> machines, to help avoid blockages due to QEMU issues.
>
> Anyway, something that's been on my mind regarding QEMU and builds is
> how well this matches up with building natively.

Here's one significant problem with emulated builds, although perhaps
not the kind of problem you were asking about:

Due to differences in the memory models of Intel and other systems such
as ARM, it is easy to write multithreaded code that works robustly on
Intel systems but sporadically fails on other architectures such as ARM.

However, an ARM emulator running on an Intel system will effectively use
an Intel memory model, because it would be prohibitively expensive to
faithfully simulate the ARM memory model on Intel hardware, and the only
benefit would be to demonstrate bugs that can't happen on Intel systems.

Therefore, test suites run under an emulator will almost certainly fail
to find architecture-specific thread safety problems, which I expect are
among the most important tests to be done, and among the most likely to
otherwise go unnoticed.

For this reason, I suggest that we should avoid emulated builds in our
build farm.

     Regards,
       Mark

Re: Implications of QEMU binfmt transparent emulation for builds

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]


Léo Le Bouter <lle-bout@zaclys.net> writes:

> On Sat, 2021-03-06 at 21:35 +0000, Christopher Baines wrote:
>
> In general I think binary translation is very wasteful use of computing
> resources. At least use something like this: 
> http://csl.iis.sinica.edu.tw/hqemu/ - otherwise I think building
> natively is at the end less costly.

Reducing waste is all about trade offs, if using inefficient emulation
means you have less hardware, then it can reduce waste.

>> I'd like the Guix Build Coordinator agents to be able to report the
>> configuration of the machine at the time builds happen, so maybe once
>> that's a feature, and QEMU support can be detected, then it'll be
>> possible to look for cases where a change in QEMU support affects the
>> build result.
>
> What about including such metadata in Nars directly? Kernel name and
> version string, system/target, processor model.. or is that too
> privacy-invasive..? Maybe make such opt-in.

There are things that rely on the hash of the nar, and adding changable
metadata would make things complicated.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implications of QEMU binfmt transparent emulation for builds

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]


Mark H Weaver <mhw@netris.org> writes:

> Hi Christopher,
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> I'm starting to play with mixing native and emulated builds with the
>> Guix Build Coordinator again. I did do this many months ago, but at that
>> time, there wasn't support for targeting retries across a range of
>> machines, to help avoid blockages due to QEMU issues.
>>
>> Anyway, something that's been on my mind regarding QEMU and builds is
>> how well this matches up with building natively.
>
> Here's one significant problem with emulated builds, although perhaps
> not the kind of problem you were asking about:
>
> Due to differences in the memory models of Intel and other systems such
> as ARM, it is easy to write multithreaded code that works robustly on
> Intel systems but sporadically fails on other architectures such as ARM.
>
> However, an ARM emulator running on an Intel system will effectively use
> an Intel memory model, because it would be prohibitively expensive to
> faithfully simulate the ARM memory model on Intel hardware, and the only
> benefit would be to demonstrate bugs that can't happen on Intel systems.
>
> Therefore, test suites run under an emulator will almost certainly fail
> to find architecture-specific thread safety problems, which I expect are
> among the most important tests to be done, and among the most likely to
> otherwise go unnoticed.
>
> For this reason, I suggest that we should avoid emulated builds in our
> build farm.

This is exactly the kind of problem I'm interested in, so thanks for the
description.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implications of QEMU binfmt transparent emulation for builds

[Ludovic Courtès]

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

> Anyway, something that's been on my mind regarding QEMU and builds is
> how well this matches up with building natively. In particular, I'm
> concerned that there are some derivations that will build on system A
> with some QEMU configuration allowing binaries for system B to be run,
> but won't build if that QEMU support wasn't there. I think there's also
> a chance that you could have a derivation for system A that builds on a
> system with QEMU support for system B, but then wouldn't build natively
> on system B.
>
> I think the first scenario is more likely, mainly because I wonder if
> this happens with cross built packages. If the package attempts to run
> software built for the target system, that will work if there's QEMU
> support there for that target system, but not if that QEMU support is
> lacking.
>
> This is just a theory at this point, I at least don't know of any cases
> of this happening, but I also haven't been looking. Any ideas?

We had concrete problems that manifested as CMake failing to find a
compiler when running emulated (emulated AArch64/ARMv7 on x86_64),
whereas it would find the compiler when running natively:

  https://issues.guix.gnu.org/43513

That’s the worst case: an issue showing up in emulated builds in a
deterministic fashion, and never happening on actual hardware.

Thanks,
Ludo’.

Re: Implications of QEMU binfmt transparent emulation for builds

[Maxim Cournoyer]

Hello Mark,

Mark H Weaver <mhw@netris.org> writes:

> Hi Christopher,
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> I'm starting to play with mixing native and emulated builds with the
>> Guix Build Coordinator again. I did do this many months ago, but at that
>> time, there wasn't support for targeting retries across a range of
>> machines, to help avoid blockages due to QEMU issues.
>>
>> Anyway, something that's been on my mind regarding QEMU and builds is
>> how well this matches up with building natively.
>
> Here's one significant problem with emulated builds, although perhaps
> not the kind of problem you were asking about:
>
> Due to differences in the memory models of Intel and other systems such
> as ARM, it is easy to write multithreaded code that works robustly on
> Intel systems but sporadically fails on other architectures such as ARM.
>
> However, an ARM emulator running on an Intel system will effectively use
> an Intel memory model, because it would be prohibitively expensive to
> faithfully simulate the ARM memory model on Intel hardware, and the only
> benefit would be to demonstrate bugs that can't happen on Intel systems.
>
> Therefore, test suites run under an emulator will almost certainly fail
> to find architecture-specific thread safety problems, which I expect are
> among the most important tests to be done, and among the most likely to
> otherwise go unnoticed.
>
> For this reason, I suggest that we should avoid emulated builds in our
> build farm.

Very interesting; thanks for sharing this great piece of information!  I
had no idea about this simplification/shortcut used by QEMU.

Maxim