bug#36117: qemu-binfmt with non-native chroot

bug#36117: qemu-binfmt with non-native chroot

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1836 bytes --]

I've been trying to use qemu-binfmt-service-type to build a non-native
chroot of a Debian system on Guix... because, well... because!

In Debian, this works with the qemu-user-static package, where the
binfmt sets these flags:

  $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
  enabled
  interpreter /usr/bin/qemu-aarch64-static
  flags: OCF
  offset 0
  magic 7f454c460201010000000000000000000200b700
  mask ffffffffffffff00fffffffffffffffffeffffff

In particular, the F flag allows the host system binaries to be used as
the interpreter inside the chroot. But apparently, this only works with
static-built qemu targets, according to the linux's
Documentation/admin-guide/binfmt-misc.rst.

On Guix there are no flags set, and the binary used is a dynamically
linked executable:

  $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
  enabled
  interpreter
  /gnu/store/sw2rrqmjij73wcy3ajd47ypvmzh12yz6-qemu-3.1.0/bin/qemu-aarch64
  flags:
  offset 0
  magic 7f454c460201010000000000000000000200b700
  mask ffffffffffffff00fffffffffffffffffeffffff


So there are (at least) two things needed to make this work on Guix:

* A way to set the flags on qemu-binfmt-service-type.

* A static build of qemu-user targets

* A way to set which qemu to use for qemu-binfmt-service-type.

The *three* things are...


With this working correctly foreign-architecture chroots would become
trivial:

  # on an amd64 host:
  $ debootstrap --arch=arm64 buster buster-chroot http://deb.debian.org/debian
  ...
  $ chroot buster-chroot /bin/bash


Enabling qemu-binfmt-service-type to operate in this way would obviate
the need for the "guix-support?" qemu-binfmt-configuration option, as
you could simply assemble the build environment without having to
include all of qemu's dependencies in the container.

It's a pretty magical feature.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#36117: qemu-binfmt with non-native chroot

[Ludovic Courtès]

Hi!

Vagrant Cascadian <vagrant@debian.org> skribis:

> On Guix there are no flags set, and the binary used is a dynamically
> linked executable:
>
>   $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
>   enabled
>   interpreter
>   /gnu/store/sw2rrqmjij73wcy3ajd47ypvmzh12yz6-qemu-3.1.0/bin/qemu-aarch64
>   flags:
>   offset 0
>   magic 7f454c460201010000000000000000000200b700
>   mask ffffffffffffff00fffffffffffffffffeffffff
>
>
> So there are (at least) two things needed to make this work on Guix:
>
> * A way to set the flags on qemu-binfmt-service-type.
>
> * A static build of qemu-user targets
>
> * A way to set which qemu to use for qemu-binfmt-service-type.
>
> The *three* things are...
>
>
> With this working correctly foreign-architecture chroots would become
> trivial:
>
>   # on an amd64 host:
>   $ debootstrap --arch=arm64 buster buster-chroot http://deb.debian.org/debian
>   ...
>   $ chroot buster-chroot /bin/bash
>
>
> Enabling qemu-binfmt-service-type to operate in this way would obviate
> the need for the "guix-support?" qemu-binfmt-configuration option, as
> you could simply assemble the build environment without having to
> include all of qemu's dependencies in the container.
>
> It's a pretty magical feature.

True!  Though adding all the dependencies of QEMU in the chroot the way
‘guix-support?’ does it turns out to be pretty magical too ;-), because
we can precisely list those dependencies and include nothing but these
dependencies in the chroot—something that cannot be done on an FHS
system.

As an quick workaround, perhaps you could bind-mount all the entries of:

  guix gc -R $(guix build qemu)

in your Debian chroot?

(Speaking of which… it would be great to have a Debian API in Guix, where
you could write, say:

  (debian-build #~(system (string-append "/bin/uname > "
                                         #$output)))

Food for thought…)

Thanks,
Ludo’.

bug#36117: qemu-binfmt with non-native chroot

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 3118 bytes --]

On 2019-06-07, Ludovic Courtès wrote:
> Vagrant Cascadian <vagrant@debian.org> skribis:
>> On Guix there are no flags set, and the binary used is a dynamically
>> linked executable:
>>
>>   $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
>>   enabled
>>   interpreter
>>   /gnu/store/sw2rrqmjij73wcy3ajd47ypvmzh12yz6-qemu-3.1.0/bin/qemu-aarch64
>>   flags:
>>   offset 0
>>   magic 7f454c460201010000000000000000000200b700
>>   mask ffffffffffffff00fffffffffffffffffeffffff
>>
>>
>> So there are (at least) two things needed to make this work on Guix:
>>
>> * A way to set the flags on qemu-binfmt-service-type.
>>
>> * A static build of qemu-user targets
>>
>> * A way to set which qemu to use for qemu-binfmt-service-type.
>>
>> The *three* things are...
>>
>>
>> With this working correctly foreign-architecture chroots would become
>> trivial:
>>
>>   # on an amd64 host:
>>   $ debootstrap --arch=arm64 buster buster-chroot http://deb.debian.org/debian
>>   ...
>>   $ chroot buster-chroot /bin/bash
>>
>>
>> Enabling qemu-binfmt-service-type to operate in this way would obviate
>> the need for the "guix-support?" qemu-binfmt-configuration option, as
>> you could simply assemble the build environment without having to
>> include all of qemu's dependencies in the container.
>>
>> It's a pretty magical feature.
>
> True!  Though adding all the dependencies of QEMU in the chroot the way
> ‘guix-support?’ does it turns out to be pretty magical too ;-), because
> we can precisely list those dependencies and include nothing but these
> dependencies in the chroot—something that cannot be done on an FHS
> system.

Indeed!


> As an quick workaround, perhaps you could bind-mount all the entries of:
>
>   guix gc -R $(guix build qemu)
>
> in your Debian chroot?

I tried an even lazier experiment, bind-mounting all of /gnu into the
new chroot directory before running debootstrap, and it worked!


That said, it's still a manual step (mounting /gnu or /gnu/store/qemu*)
required to do something that could otherwise be handled transparently
with a static build of qemu and adjusting the binfmt_misc flags... so if
permitted to dream, I still think that would be a nice option to have
available. :)


Another interesting angle is that including qemu and all of qemu's
dependencies in a guix build environment is that qemu or one of it's
dependencies might actually get used during the build... even if not
explicitly included in one of the inputs or one of the build systems. So
maybe the case can be made that the qemu-static build from executed from
the host system is cleaner than copying all of qemu and dependencies
into the build environment...


> (Speaking of which… it would be great to have a Debian API in Guix, where
> you could write, say:
>
>   (debian-build #~(system (string-append "/bin/uname > "
>                                          #$output)))
>
> Food for thought…)

Not quite sure of what you're going for, but perhaps best to have that
conversation elsewhere.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#36117: qemu-binfmt with non-native chroot

[Ludovic Courtès]

Hi!

Vagrant Cascadian <vagrant@debian.org> skribis:

> That said, it's still a manual step (mounting /gnu or /gnu/store/qemu*)
> required to do something that could otherwise be handled transparently
> with a static build of qemu and adjusting the binfmt_misc flags... so if
> permitted to dream, I still think that would be a nice option to have
> available. :)

Yes, it sounds like we could/should do that too!

We’ll end up having big qemu-* binaries, each one with its own copy of
glibc, for instance.  Is there something smart we could do to avoid
ending up with a huge package?

Thanks,
Ludo’.

bug#36117: [PATCH 1/4] gnu: qemu: Fix indentation and remove trailing #t.

[Maxim Cournoyer]

* gnu/packages/virtualization.scm (qemu): Fix indentation and remove
trailing #t.
---
 gnu/packages/virtualization.scm | 45 +++++++++++++++------------------
 1 file changed, 21 insertions(+), 24 deletions(-)

diff --git a/gnu/packages/virtualization.scm b/gnu/packages/virtualization.scm
index 8da57cf6ab..5d506cbf54 100644
--- a/gnu/packages/virtualization.scm
+++ b/gnu/packages/virtualization.scm
@@ -138,9 +138,9 @@
               (method url-fetch)
               (uri (string-append "https://download.qemu.org/qemu-"
                                   version ".tar.xz"))
-               (sha256
-                (base32
-                 "1rd41wwlvp0vpialjp2czs6i3lsc338xc72l3zkbb7ixjfslw5y9"))
+              (sha256
+               (base32
+                "1rd41wwlvp0vpialjp2czs6i3lsc338xc72l3zkbb7ixjfslw5y9"))
               (patches (search-patches "qemu-build-info-manual.patch"))
               (modules '((guix build utils)))
               (snippet
@@ -152,14 +152,16 @@
                   ;; https://lists.nongnu.org/archive/html/qemu-devel/2021-01/msg01545.html
                   (substitute* '("linux-user/syscall.c")
                     (("^([[:blank:]]*)const argtype ifreq_arg_type.*$" line indent)
-                     (string-append line indent
-                                    "const argtype ifreq_max_type[] = { MK_STRUCT(STRUCT_ifmap_ifreq) };\n"))
+                     (string-append
+                      line indent
+                      "const argtype ifreq_max_type[] = { MK_STRUCT(STRUCT_ifmap_ifreq) };\n"))
                     (("^([[:blank:]]*)target_ifreq_size[[:blank:]]=.*$" _ indent)
-                     (string-append indent "target_ifreq_size = thunk_type_size(ifreq_max_type, 0);")))
-                  #t))))
-     (outputs '("out" "doc"))            ;4.7 MiB of HTML docs
-     (build-system gnu-build-system)
-     (arguments
+                     (string-append
+                      indent
+                      "target_ifreq_size = thunk_type_size(ifreq_max_type, 0);")))))))
+    (outputs '("out" "doc"))            ;4.7 MiB of HTML docs
+    (build-system gnu-build-system)
+    (arguments
      `(;; FIXME: Disable tests on i686 to work around
        ;; <https://bugs.gnu.org/40527>.
        #:tests? ,(or (%current-target-system)
@@ -193,8 +195,7 @@
                                  inputs)))
                (set-path-environment-variable "C_INCLUDE_PATH"
                                               '("include")
-                                              input-directories)
-               #t)))
+                                              input-directories))))
          (add-after 'unpack 'extend-test-time-outs
            (lambda _
              ;; These tests can time out on heavily-loaded and/or slow storage.
@@ -218,16 +219,14 @@
                ;; x86 CPUs (see https://issues.guix.info/43048 and
                ;; https://bugs.launchpad.net/qemu/+bug/1896263).
                (("check-qtest-i386-y \\+= bios-tables-test" all)
-                (string-append "# " all)))
-             #t))
+                (string-append "# " all)))))
          (add-after 'patch-source-shebangs 'patch-/bin/sh-references
            (lambda _
              ;; Ensure the executables created by these source files reference
              ;; /bin/sh from the store so they work inside the build container.
              (substitute* '("block/cloop.c" "migration/exec.c"
                             "net/tap.c" "tests/qtest/libqtest.c")
-               (("/bin/sh") (which "sh")))
-             #t))
+               (("/bin/sh") (which "sh")))))
          (replace 'configure
            (lambda* (#:key inputs outputs (configure-flags '())
                      #:allow-other-keys)
@@ -272,8 +271,7 @@
                    (format port "#!/bin/sh
 exec smbd $@")))
                (chmod "samba-wrapper" #o755)
-               (install-file "samba-wrapper" libexec))
-             #t))
+               (install-file "samba-wrapper" libexec))))
          (add-after 'install 'move-html-doc
            (lambda* (#:key inputs outputs #:allow-other-keys)
              (let* ((out (assoc-ref outputs "out"))
@@ -281,23 +279,22 @@ exec smbd $@")))
                     (qemu-doc (string-append doc "/share/doc/qemu-" ,version)))
                (mkdir-p qemu-doc)
                (rename-file (string-append out "/share/doc/qemu")
-                            (string-append qemu-doc "/html")))
-             #t)))))
-    (inputs                                       ; TODO: Add optional inputs.
+                            (string-append qemu-doc "/html"))))))))
+    (inputs                             ; TODO: Add optional inputs.
      `(("alsa-lib" ,alsa-lib)
        ("attr" ,attr)
        ("glib" ,glib)
        ("gtk+" ,gtk+)
        ("libaio" ,libaio)
        ("libattr" ,attr)
-       ("libcacard" ,libcacard)     ; smartcard support
-       ("libcap-ng" ,libcap-ng)     ; virtfs support requires libcap-ng & libattr
+       ("libcacard" ,libcacard)  ; smartcard support
+       ("libcap-ng" ,libcap-ng)  ; virtfs support requires libcap-ng & libattr
        ("libdrm" ,libdrm)
        ("libepoxy" ,libepoxy)
        ("libjpeg" ,libjpeg-turbo)
        ("libpng" ,libpng)
        ("libseccomp" ,libseccomp)
-       ("libusb" ,libusb)                         ;USB pass-through support
+       ("libusb" ,libusb)               ;USB pass-through support
        ("mesa" ,mesa)
        ("ncurses" ,ncurses)
        ;; ("pciutils" ,pciutils)
-- 
2.30.1

bug#36117: [PATCH 2/4] gnu: Add glib-static.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 2152 bytes --]

* gnu/packages/glib.scm (glib-static): New variable.
---
 gnu/packages/glib.scm | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/glib.scm b/gnu/packages/glib.scm
index 9cc2b1b69e..18c905b70d 100644
--- a/gnu/packages/glib.scm
+++ b/gnu/packages/glib.scm
@@ -9,7 +9,7 @@
 ;;; Copyright © 2017 Petter <petter@mykolab.ch>
 ;;; Copyright © 2018, 2019 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
-;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2019, 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2019 Giacomo Leidi <goodoldpaul@autistici.org>
 ;;; Copyright © 2019, 2020 Marius Bakke <mbakke@fastmail.com>
 ;;; Copyright © 2020 Nicolò Balzarotti <nicolo@nixo.xyz>
@@ -417,6 +417,34 @@ dynamic loading, and an object system.")
                  (delete-file-recursively (string-append out html))
                  #t)))))))))
 
+;;; TODO: Merge into glib as a 'static' output in core-updates.
+(define-public glib-static
+  (hidden-package
+   (package
+     (inherit glib)
+     (name "glib-static")
+     (outputs '("out"))
+     (arguments
+      (substitute-keyword-arguments (package-arguments glib)
+        ((#:configure-flags flags ''())
+         `(cons* "--default-library=static"
+                 "-Dselinux=disabled"
+                 "-Dman=false"
+                 "-Dgtk_doc=false"
+                 "-Dinternal_pcre=false"
+                 ,flags))
+        ((#:phases phases)
+         `(modify-phases ,phases
+            (delete 'move-executables)
+            (replace 'install
+              ;; Only install the static libraries.
+              (lambda* (#:key outputs #:allow-other-keys)
+                (let* ((out (assoc-ref outputs "out"))
+                       (lib (string-append out "/lib")))
+                  (for-each (lambda (f)
+                              (install-file f lib))
+                            (find-files "." "\\.a$"))))))))))))
+
 (define gobject-introspection
   (package
     (name "gobject-introspection")
-- 
2.30.1

bug#36117: [PATCH 3/4] gnu: qemu: Add a static output.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 13962 bytes --]

The static output is equivalent to what other distributions commonly package
as 'qemu-user-static'.

* gnu/packages/virtualization.scm (qemu)[outputs]: Add a static output.
[phases]{configure}: Configure the main build as an out-of-source build.  Move
all configure flags to ...
[configure-flags]: ... here.  The options explicitly enabling optional
features are removed; the configure script does a good job at enabling all the
features available based on the inputs present and this allows reusing the
flags in variant packages such as qemu-minimal.
{configure-user-static, build-user-static, install-user-static}: New phases.
{patch-test-shebangs}: New phase, extracted from the configure phase.
[native-inputs]: Add glib-static, pcre:static and zlib:static.
(qemu-minimal)[arguments]: Reuse the configure-flags argument.  Rewrite to use
match instead of cond.
---
 gnu/packages/virtualization.scm | 170 ++++++++++++++++++++------------
 1 file changed, 105 insertions(+), 65 deletions(-)

diff --git a/gnu/packages/virtualization.scm b/gnu/packages/virtualization.scm
index 5d506cbf54..b1cebeb7f8 100644
--- a/gnu/packages/virtualization.scm
+++ b/gnu/packages/virtualization.scm
@@ -15,7 +15,7 @@
 ;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re>
 ;;; Copyright © 2020 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;; Copyright © 2020 Marius Bakke <mbakke@fastmail.com>
-;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2020, 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2020 Brett Gilio <brettg@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -83,6 +83,7 @@
   #:use-module (gnu packages onc-rpc)
   #:use-module (gnu packages package-management)
   #:use-module (gnu packages perl)
+  #:use-module (gnu packages pcre)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages polkit)
   #:use-module (gnu packages protobuf)
@@ -118,6 +119,7 @@
   #:use-module (guix packages)
   #:use-module (guix utils)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:use-module (ice-9 match))
 
 (define (qemu-patch commit file-name sha256-bv)
@@ -159,23 +161,30 @@
                      (string-append
                       indent
                       "target_ifreq_size = thunk_type_size(ifreq_max_type, 0);")))))))
-    (outputs '("out" "doc"))            ;4.7 MiB of HTML docs
+    (outputs '("out" "static" "doc"))   ;4.7 MiB of HTML docs
     (build-system gnu-build-system)
     (arguments
-     `(;; FIXME: Disable tests on i686 to work around
-       ;; <https://bugs.gnu.org/40527>.
-       #:tests? ,(or (%current-target-system)
+     ;; FIXME: Disable tests on i686 to work around
+     ;; <https://bugs.gnu.org/40527>.
+     `(#:tests? ,(or (%current-target-system)
                      (not (string=? "i686-linux" (%current-system))))
-
-       #:configure-flags (list "--enable-usb-redir" "--enable-opengl"
-                               "--enable-docs"
-                               (string-append "--smbd="
-                                              (assoc-ref %outputs "out")
-                                              "/libexec/samba-wrapper")
-                               "--audio-drv-list=alsa,pa,sdl")
+       #:configure-flags
+       (let ((gcc (string-append (assoc-ref %build-inputs "gcc") "/bin/gcc"))
+             (out (assoc-ref %outputs "out")))
+         (list (string-append "--cc=" gcc)
+               ;; Some architectures insist on using HOST_CC.
+               (string-append "--host-cc=" gcc)
+               (string-append "--prefix=" out)
+               "--sysconfdir=/etc"
+               (string-append "--smbd=" out "/libexec/samba-wrapper")
+               "--disable-debug-info"   ;for space considerations
+               ;; The binaries need to be linked against -lrt.
+               (string-append "--extra-ldflags=-lrt")))
        ;; Make build and test output verbose to facilitate investigation upon failure.
        #:make-flags '("V=1")
        #:modules ((srfi srfi-1)
+                  (srfi srfi-26)
+                  (ice-9 ftw)
                   (ice-9 match)
                   ,@%gnu-build-system-modules)
        #:phases
@@ -220,6 +229,11 @@
                ;; https://bugs.launchpad.net/qemu/+bug/1896263).
                (("check-qtest-i386-y \\+= bios-tables-test" all)
                 (string-append "# " all)))))
+         (add-after 'unpack 'patch-test-shebangs
+           (lambda _
+             (substitute* "tests/qemu-iotests/check"
+               (("#!/usr/bin/env python3")
+                (string-append "#!" (which "python3"))))))
          (add-after 'patch-source-shebangs 'patch-/bin/sh-references
            (lambda _
              ;; Ensure the executables created by these source files reference
@@ -228,37 +242,55 @@
                             "net/tap.c" "tests/qtest/libqtest.c")
                (("/bin/sh") (which "sh")))))
          (replace 'configure
-           (lambda* (#:key inputs outputs (configure-flags '())
-                     #:allow-other-keys)
+           (lambda* (#:key inputs outputs configure-flags #:allow-other-keys)
              ;; The `configure' script doesn't understand some of the
              ;; GNU options.  Thus, add a new phase that's compatible.
              (let ((out (assoc-ref outputs "out")))
                (setenv "SHELL" (which "bash"))
-
-               ;; While we're at it, patch for tests.
-               (substitute* "tests/qemu-iotests/check"
-                 (("#!/usr/bin/env python3")
-                  (string-append "#!" (which "python3"))))
-
                ;; Ensure config.status gets the correct shebang off the bat.
                ;; The build system gets confused if we change it later and
                ;; attempts to re-run the whole configury, and fails.
                (substitute* "configure"
                  (("#!/bin/sh")
                   (string-append "#!" (which "sh"))))
-
-               ;; The binaries need to be linked against -lrt.
-               (setenv "LDFLAGS" "-lrt")
-               (apply invoke
-                      `("./configure"
-                        ,(string-append "--cc=" (which "gcc"))
-                        ;; Some architectures insist on using HOST_CC
-                        ,(string-append "--host-cc=" (which "gcc"))
-                        "--disable-debug-info" ; save build space
-                        "--enable-virtfs"      ; just to be sure
-                        ,(string-append "--prefix=" out)
-                        ,(string-append "--sysconfdir=/etc")
-                        ,@configure-flags)))))
+               (mkdir-p "b/qemu")
+               (chdir "b/qemu")
+               (apply invoke "../../configure" configure-flags))))
+         ;; Configure, build and install QEMU user-emulation static binaries.
+         (add-after 'configure 'configure-user-static
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let* ((gcc (string-append (assoc-ref inputs "gcc") "/bin/gcc"))
+                    (static (assoc-ref outputs "static"))
+                    ;; This is the common set of configure flags; it is
+                    ;; duplicated here to isolate this phase from manipulations
+                    ;; to the #:configure-flags build argument, as done in
+                    ;; derived packages such as qemu-minimal.
+                    (configure-flags (list (string-append "--cc=" gcc)
+                                           (string-append "--host-cc=" gcc)
+                                           "--sysconfdir=/etc"
+                                           "--disable-debug-info")))
+               (mkdir-p "../user-static")
+               (with-directory-excursion "../user-static"
+                 (apply invoke "../../configure"
+                        "--static"
+                        "--disable-system"
+                        "--enable-linux-user"
+                        (string-append "--prefix=" static)
+                        configure-flags)))))
+         (add-after 'build 'build-user-static
+           (lambda args
+             (with-directory-excursion "../user-static"
+               (apply (assoc-ref %standard-phases 'build) args))))
+         (add-after 'install 'install-user-static
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((static (assoc-ref outputs "static"))
+                    (bin (string-append static "/bin")))
+               (with-directory-excursion "../user-static"
+                 (for-each (cut install-file <> bin)
+                           (append-map (cut find-files <> "^qemu-")
+                                       (scandir "."
+                                                (cut string-suffix?
+                                                     "-linux-user" <>))))))))
          ;; Create a wrapper for Samba. This allows QEMU to use Samba without
          ;; pulling it in as an input. Note that you need to explicitly install
          ;; Samba in your Guix profile for Samba support.
@@ -315,7 +347,12 @@ exec smbd $@")))
                      ("pkg-config" ,pkg-config)
                      ("python-wrapper" ,python-wrapper)
                      ("python-sphinx" ,python-sphinx)
-                     ("texinfo" ,texinfo)))
+                     ("texinfo" ,texinfo)
+                     ;; The following static libraries are required to build
+                     ;; the static output of QEMU.
+                     ("glib-static" ,glib-static)
+                     ("pcre:static" ,pcre "static")
+                     ("zlib:static" ,zlib "static")))
     (home-page "https://www.qemu.org")
     (synopsis "Machine emulator and virtualizer")
     (description
@@ -340,46 +377,49 @@ server and embedded PowerPC, and S390 guests.")
 
 (define-public qemu-minimal
   ;; QEMU without GUI support, only supporting the host's architecture
-  (package (inherit qemu)
+  (package
+    (inherit qemu)
     (name "qemu-minimal")
     (synopsis
      "Machine emulator and virtualizer (without GUI) for the host architecture")
     (arguments
      (substitute-keyword-arguments (package-arguments qemu)
-       ((#:configure-flags _ '(list))
+       ((#:configure-flags configure-flags '(list))
         ;; Restrict to the host's architecture.
-        (let ((system (or (%current-target-system)
-                          (%current-system))))
-          (cond
-            ((string-prefix? "i686" system)
-             '(list "--target-list=i386-softmmu"))
-            ((string-prefix? "xasdf86_64" system)
-             '(list "--target-list=i386-softmmu,x86_64-softmmu"))
-            ((string-prefix? "mips64" system)
-             '(list (string-append "--target-list=mips-softmmu,mipsel-softmmu,"
-                                   "mips64-softmmu,mips64el-softmmu")))
-            ((string-prefix? "mips" system)
-             '(list "--target-list=mips-softmmu,mipsel-softmmu"))
-            ((string-prefix? "aarch64" system)
-             '(list "--target-list=arm-softmmu,aarch64-softmmu"))
-            ((string-prefix? "arm" system)
-             '(list "--target-list=arm-softmmu"))
-            ((string-prefix? "alpha" system)
-             '(list "--target-list=alpha-softmmu"))
-            ((string-prefix? "powerpc64" system)
-             '(list "--target-list=ppc-softmmu,ppc64-softmmu"))
-            ((string-prefix? "powerpc" system)
-             '(list "--target-list=ppc-softmmu"))
-            ((string-prefix? "s390" system)
-             '(list "--target-list=s390x-softmmu"))
-            ((string-prefix? "riscv" system)
-             '(list "--target-list=riscv32-softmmu,riscv64-softmmu"))
-            (else   ; An empty list actually builds all the targets.
-              ''()))))))
+        (let* ((system (or (%current-target-system)
+                           (%current-system)))
+               (target-list-arg
+                (match system
+                  ((? (cut string-prefix? "i686" <>))
+                   "--target-list=i386-softmmu")
+                  ((? (cut string-prefix? "x86_64" <>))
+                   "--target-list=i386-softmmu,x86_64-softmmu")
+                  ((? (cut string-prefix? "mips64" <>))
+                   (string-append "--target-list=mips-softmmu,mipsel-softmmu,"
+                                  "mips64-softmmu,mips64el-softmmu"))
+                  ((? (cut string-prefix? "mips" <>))
+                   "--target-list=mips-softmmu,mipsel-softmmu")
+                  ((? (cut string-prefix? "aarch64" <>))
+                   "--target-list=arm-softmmu,aarch64-softmmu")
+                  ((? (cut string-prefix? "arm" <>))
+                   "--target-list=arm-softmmu")
+                  ((? (cut string-prefix? "alpha" <>))
+                   "--target-list=alpha-softmmu")
+                  ((? (cut string-prefix? "powerpc64" <>))
+                   "--target-list=ppc-softmmu,ppc64-softmmu")
+                  ((? (cut string-prefix? "powerpc" <>))
+                   "--target-list=ppc-softmmu")
+                  ((? (cut string-prefix? "s390" <>))
+                   "--target-list=s390x-softmmu")
+                  ((? (cut string-prefix? "riscv" <>))
+                   "--target-list=riscv32-softmmu,riscv64-softmmu")
+                  (else       ; An empty list actually builds all the targets.
+                   '()))))
+          `(cons ,target-list-arg ,configure-flags)))))
 
     ;; Remove dependencies on optional libraries, notably GUI libraries.
     (native-inputs (fold alist-delete (package-native-inputs qemu)
-                  '("gettext")))
+                         '("gettext")))
     (inputs (fold alist-delete (package-inputs qemu)
                   '("libusb" "mesa" "sdl2" "spice" "virglrenderer" "gtk+"
                     "usbredir" "libdrm" "libepoxy" "pulseaudio" "vde2"
-- 
2.30.1

bug#36117: [PATCH 3/4] gnu: qemu: Add a static output.

[Ludovic Courtès]

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> The static output is equivalent to what other distributions commonly package
> as 'qemu-user-static'.
>
> * gnu/packages/virtualization.scm (qemu)[outputs]: Add a static output.
> [phases]{configure}: Configure the main build as an out-of-source build.  Move
> all configure flags to ...
> [configure-flags]: ... here.  The options explicitly enabling optional
> features are removed; the configure script does a good job at enabling all the
> features available based on the inputs present and this allows reusing the
> flags in variant packages such as qemu-minimal.
> {configure-user-static, build-user-static, install-user-static}: New phases.
> {patch-test-shebangs}: New phase, extracted from the configure phase.
> [native-inputs]: Add glib-static, pcre:static and zlib:static.
> (qemu-minimal)[arguments]: Reuse the configure-flags argument.  Rewrite to use
> match instead of cond.

Nice!

Do you know how this affects build times?  An option would be to make a
separate package rather than a separate output, if that helps.

Thanks,
Ludo’.

bug#36117: [PATCH 3/4] gnu: qemu: Add a static output.

[Maxim Cournoyer]

Hi!

Ludovic Courtès <ludo@gnu.org> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> The static output is equivalent to what other distributions commonly package
>> as 'qemu-user-static'.
>>
>> * gnu/packages/virtualization.scm (qemu)[outputs]: Add a static output.
>> [phases]{configure}: Configure the main build as an out-of-source build.  Move
>> all configure flags to ...
>> [configure-flags]: ... here.  The options explicitly enabling optional
>> features are removed; the configure script does a good job at enabling all the
>> features available based on the inputs present and this allows reusing the
>> flags in variant packages such as qemu-minimal.
>> {configure-user-static, build-user-static, install-user-static}: New phases.
>> {patch-test-shebangs}: New phase, extracted from the configure phase.
>> [native-inputs]: Add glib-static, pcre:static and zlib:static.
>> (qemu-minimal)[arguments]: Reuse the configure-flags argument.  Rewrite to use
>> match instead of cond.
>
> Nice!
>
> Do you know how this affects build times?  An option would be to make a
> separate package rather than a separate output, if that helps.
>
> Thanks,
> Ludo’.

I've just ran two builds; the current qemu we have, and the one with
the static output added:

/gnu/store/aqkr351pph30za9v6bmd8gzpq5gbgz0c-qemu-5.1.0-doc
/gnu/store/apxxj92gncx901z71kq0gwg48n2cils6-qemu-5.1.0

real    6m56.816s
user    0m13.733s
sys     0m0.677s

/gnu/store/4bfj6zxl0yd2i3206v4rkwiv2nwyryxb-qemu-5.1.0-doc
/gnu/store/7w04gv6m92n40dainn4s6xr3l20r90xw-qemu-5.1.0
/gnu/store/wqh2dyskzkl4vjn6harclyl317h4vfaf-qemu-5.1.0-static

real    7m39.197s
user    0m15.035s
sys     0m0.841s

That's about a 10% increase in build time, which is reasonable IMO.

In the future we should definitely split our QEMU package into
per-architecture variants as well as user vs system emulation, similar
to what Debian does.  A near 755.8 MiB package is cumbersome and I'm
doubtful most of its users make use of the 35 architectures it boasts at
a time :-).

Thank you!

Maxim

bug#36117: [PATCH 3/4] gnu: qemu: Add a static output.

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> I've just ran two builds; the current qemu we have, and the one with
> the static output added:
>
> /gnu/store/aqkr351pph30za9v6bmd8gzpq5gbgz0c-qemu-5.1.0-doc
> /gnu/store/apxxj92gncx901z71kq0gwg48n2cils6-qemu-5.1.0
>
> real    6m56.816s
> user    0m13.733s
> sys     0m0.677s
>
> /gnu/store/4bfj6zxl0yd2i3206v4rkwiv2nwyryxb-qemu-5.1.0-doc
> /gnu/store/7w04gv6m92n40dainn4s6xr3l20r90xw-qemu-5.1.0
> /gnu/store/wqh2dyskzkl4vjn6harclyl317h4vfaf-qemu-5.1.0-static
>
> real    7m39.197s
> user    0m15.035s
> sys     0m0.841s
>
> That's about a 10% increase in build time, which is reasonable IMO.

Yes, sounds good.

> In the future we should definitely split our QEMU package into
> per-architecture variants as well as user vs system emulation, similar
> to what Debian does.  A near 755.8 MiB package is cumbersome and I'm
> doubtful most of its users make use of the 35 architectures it boasts at
> a time :-).

Agreed!

Thanks,
Ludo’.

bug#36117: [PATCH 4/4] services/qemu-binfmt: Use the F flag and the static output of QEMU.

[Maxim Cournoyer]

Fixes <https://issues.guix.gnu.org/36117>.

Before this change, the 'binfmt_misc' entries registered for QEMU would not be
usable in container contexts outside of guix-daemon (without manually bind
mounting file names).

For example:

$ docker run --rm arm32v7/debian true
standard_init_linux.go:207: exec user process caused "no such file or directory"

After this change, any container can make use of the QEMU binfmt_misc
registrations, as their corresponding QEMU static binaries are fully
pre-loaded by the kernel.

* gnu/services/virtualization.scm (<qemu-platform>): Define using
'define-record-type*'.
[flags]: New field, which defaults to "F" (fix binary).
(%i386, %i486, %alpha, %arm, %armeb, %sparc, %sparc32plus, %ppc, %ppc64)
(%ppc64le, %m68k, %mips, %mipsel, %mipsn32, %mipsn32el, %mips64, %mips64el)
(%riscv32, %riscv64, %sh4, %sh4eb, %s390x, %aarch64, %hppa): Adjust.
(qemu-binfmt-guix-chroot): Remove variable.
(qemu-binfmt-service-type): Remove the qemu-binfmt-guix-chroot extension.
* gnu/services/qemu-binfmt (qemu-platform->binfmt):  Use the static output of
QEMU.
* doc/contributing.texi (Submitting Patches): Update doc.
* doc/guix.texi (Virtualization Services): Update doc.
---
 doc/contributing.texi           |   1 -
 doc/guix.texi                   |  12 --
 gnu/services/virtualization.scm | 227 ++++++++++++++++++--------------
 3 files changed, 129 insertions(+), 111 deletions(-)

diff --git a/doc/contributing.texi b/doc/contributing.texi
index 3baedb0654..94fb22316d 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -1016,7 +1016,6 @@ your @code{operating-system} configuration:
 (service qemu-binfmt-service-type
  (qemu-binfmt-configuration
    (platforms (lookup-qemu-platforms "arm" "aarch64"))
-   (guix-support? #t)))
 @end lisp
 
 Then reconfigure your system.
diff --git a/doc/guix.texi b/doc/guix.texi
index 5f55853159..7a4340a1d9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -28645,13 +28645,6 @@ This is the configuration for the @code{qemu-binfmt} service.
 The list of emulated QEMU platforms.  Each item must be a @dfn{platform
 object} as returned by @code{lookup-qemu-platforms} (see below).
 
-@item @code{guix-support?} (default: @code{#t})
-When it is true, QEMU and all its dependencies are added to the build
-environment of @command{guix-daemon} (@pxref{Invoking guix-daemon,
-@option{--chroot-directory} option}).  This allows the @code{binfmt_misc}
-handlers to be used within the build environment, which in turn means
-that you can transparently build programs for another architecture.
-
 For example, let's suppose you're on an x86_64 machine and you have this
 service:
 
@@ -28659,7 +28652,6 @@ service:
 (service qemu-binfmt-service-type
          (qemu-binfmt-configuration
            (platforms (lookup-qemu-platforms "arm"))
-           (guix-support? #t)))
 @end lisp
 
 You can run:
@@ -28674,10 +28666,6 @@ build}, transparently using QEMU to emulate the ARMv7 CPU@.  Pretty handy
 if you'd like to test a package build for an architecture you don't have
 access to!
 
-When @code{guix-support?} is set to @code{#f}, programs for other
-architectures can still be executed transparently, but invoking commands
-like @command{guix build -s armhf-linux @dots{}} will fail.
-
 @item @code{qemu} (default: @code{qemu})
 The QEMU package to use.
 @end table
diff --git a/gnu/services/virtualization.scm b/gnu/services/virtualization.scm
index a45da14a80..36e9feb05c 100644
--- a/gnu/services/virtualization.scm
+++ b/gnu/services/virtualization.scm
@@ -554,13 +554,14 @@ potential infinite waits blocking libvirt."))
 ;;;
 
 ;; Platforms that QEMU can emulate.
-(define-record-type <qemu-platform>
-  (qemu-platform name family magic mask)
+(define-record-type* <qemu-platform>
+  qemu-platform make-qemu-platform
   qemu-platform?
   (name     qemu-platform-name)                   ;string
   (family   qemu-platform-family)                 ;string
   (magic    qemu-platform-magic)                  ;bytevector
-  (mask     qemu-platform-mask))                  ;bytevector
+  (mask     qemu-platform-mask)                   ;bytevector
+  (flags    qemu-platform-flags (default "F")))   ;string
 
 (define-syntax bv
   (lambda (s)
@@ -577,125 +578,173 @@ potential infinite waits blocking libvirt."))
 ;;; 'scripts/qemu-binfmt-conf.sh' in QEMU.
 
 (define %i386
-  (qemu-platform "i386" "i386"
-                 (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00")
-                 (bv "\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "i386")
+   (family "i386")
+   (magic (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %i486
-  (qemu-platform "i486" "i386"
-                 (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x06\x00")
-                 (bv "\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "i486")
+   (family "i386")
+   (magic (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x06\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %alpha
-  (qemu-platform "alpha" "alpha"
-                 (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90")
-                 (bv "\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "alpha")
+   (family "alpha")
+   (magic (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90"))
+   (mask (bv "\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %arm
-  (qemu-platform "arm" "arm"
-                 (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "arm")
+   (family "arm")
+   (magic (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %armeb
-  (qemu-platform "armeb" "arm"
-                 (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "armeb")
+   (family "arm")
+   (magic (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %sparc
-  (qemu-platform "sparc" "sparc"
-                 (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "sparc")
+   (family "sparc")
+   (magic (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %sparc32plus
-  (qemu-platform "sparc32plus" "sparc"
-                 (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x12")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "sparc32plus")
+   (family "sparc")
+   (magic (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x12"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %ppc
-  (qemu-platform "ppc" "ppc"
-                 (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "ppc")
+   (family "ppc")
+   (magic (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %ppc64
-  (qemu-platform "ppc64" "ppc"
-                 (bv "\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "ppc64")
+   (family "ppc")
+   (magic (bv "\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %ppc64le
-  (qemu-platform "ppc64le" "ppcle"
-                 (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\x00")))
+  (qemu-platform
+   (name "ppc64le")
+   (family "ppcle")
+   (magic (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\x00"))))
 
 (define %m68k
-  (qemu-platform "m68k" "m68k"
-                 (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04")
-                 (bv "\xff\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "m68k")
+   (family "m68k")
+   (magic (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 ;; XXX: We could use the other endianness on a MIPS host.
 (define %mips
-  (qemu-platform "mips" "mips"
-                 (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "mips")
+   (family "mips")
+   (magic (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %mipsel
-  (qemu-platform "mipsel" "mips"
-                 (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "mipsel")
+   (family "mips")
+   (magic (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %mipsn32
-  (qemu-platform "mipsn32" "mips"
-                 (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "mipsn32")
+   (family "mips")
+   (magic (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %mipsn32el
-  (qemu-platform "mipsn32el" "mips"
-                 (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "mipsn32el")
+   (family "mips")
+   (magic (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %mips64
-  (qemu-platform "mips64" "mips"
-                 (bv "\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "mips64")
+   (family "mips")
+   (magic (bv "\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %mips64el
-  (qemu-platform "mips64el" "mips"
-                 (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "mips64el")
+   (family "mips")
+   (magic (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %riscv32
-  (qemu-platform "riscv32" "riscv"
-                 (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "riscv32")
+   (family "riscv")
+   (magic (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %riscv64
-  (qemu-platform "riscv64" "riscv"
-                 (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "riscv64")
+   (family "riscv")
+   (magic (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %sh4
-  (qemu-platform "sh4" "sh4"
-                 (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "sh4")
+   (family "sh4")
+   (magic (bv "\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %sh4eb
-  (qemu-platform "sh4eb" "sh4"
-                 (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "sh4eb")
+   (family "sh4")
+   (magic (bv "\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %s390x
-  (qemu-platform "s390x" "s390x"
-                 (bv "\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x16")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "s390x")
+   (family "s390x")
+   (magic (bv "\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x16"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %aarch64
-  (qemu-platform "aarch64" "arm"
-                 (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff")))
+  (qemu-platform
+   (name "aarch64")
+   (family "arm")
+   (magic (bv "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"))))
 
 (define %hppa
-  (qemu-platform "hppa" "hppa"
-                 (bv "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x0f")
-                 (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff")))
+  (qemu-platform
+   (name "hppa")
+   (family "hppa")
+   (magic (bv "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x0f"))
+   (mask (bv "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff"))))
 
 (define %qemu-platforms
   (list %i386 %i486 %alpha %arm %sparc32plus %ppc %ppc64 %ppc64le %m68k
@@ -715,9 +764,7 @@ potential infinite waits blocking libvirt."))
   (qemu        qemu-binfmt-configuration-qemu
                (default qemu))
   (platforms   qemu-binfmt-configuration-platforms
-               (default '()))                     ;safest default
-  (guix-support? qemu-binfmt-configuration-guix-support?
-                 (default #t)))
+               (default '())))          ;safest default
 
 (define (qemu-platform->binfmt qemu platform)
   "Return a gexp that evaluates to a binfmt string for PLATFORM, using the
@@ -733,14 +780,13 @@ given QEMU package."
           (bytevector->u8-list bv))))
 
   (match platform
-    (($ <qemu-platform> name family magic mask)
+    (($ <qemu-platform> name family magic mask flags)
      ;; See 'Documentation/binfmt_misc.txt' in the kernel.
      #~(string-append ":qemu-" #$name ":M::"
                       #$(bytevector->binfmt-string magic)
                       ":" #$(bytevector->binfmt-string mask)
-                      ":" #$(file-append qemu "/bin/qemu-" name)
-                      ":"                         ;FLAGS go here
-                      ))))
+                      ":" #$qemu:static "/bin/qemu-" #$name
+                      ":" #$flags))))
 
 (define %binfmt-mount-point
   (file-system-mount-point %binary-format-file-system))
@@ -779,19 +825,6 @@ given QEMU package."
                                 '#$(map qemu-platform-name platforms))
                       #f)))))))
 
-(define qemu-binfmt-guix-chroot
-  (match-lambda
-    ;; Add QEMU and its dependencies to the guix-daemon chroot so that our
-    ;; binfmt_misc handlers work in the chroot (otherwise 'execve' would fail
-    ;; with ENOENT.)
-    ;;
-    ;; The 'F' flag of binfmt_misc is meant to address this problem by loading
-    ;; the interpreter upfront rather than lazily, but apparently that is
-    ;; insufficient (perhaps it loads the 'qemu-ARCH' binary upfront but looks
-    ;; up its dependencies lazily?).
-    (($ <qemu-binfmt-configuration> qemu platforms guix?)
-     (if guix? (list qemu) '()))))
-
 (define qemu-binfmt-service-type
   ;; TODO: Make a separate binfmt_misc service out of this?
   (service-type (name 'qemu-binfmt)
@@ -800,9 +833,7 @@ given QEMU package."
                                           (const
                                            (list %binary-format-file-system)))
                        (service-extension shepherd-root-service-type
-                                          qemu-binfmt-shepherd-services)
-                       (service-extension guix-service-type
-                                          qemu-binfmt-guix-chroot)))
+                                          qemu-binfmt-shepherd-services)))
                 (default-value (qemu-binfmt-configuration))
                 (description
                  "This service supports transparent emulation of binaries
-- 
2.30.1

bug#36117: [PATCH 4/4] services/qemu-binfmt: Use the F flag and the static output of QEMU.

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Fixes <https://issues.guix.gnu.org/36117>.
>
> Before this change, the 'binfmt_misc' entries registered for QEMU would not be
> usable in container contexts outside of guix-daemon (without manually bind
> mounting file names).
>
> For example:
>
> $ docker run --rm arm32v7/debian true
> standard_init_linux.go:207: exec user process caused "no such file or directory"
>
> After this change, any container can make use of the QEMU binfmt_misc
> registrations, as their corresponding QEMU static binaries are fully
> pre-loaded by the kernel.

[...]

> +++ b/gnu/services/virtualization.scm
> @@ -554,13 +554,14 @@ potential infinite waits blocking libvirt."))
>  ;;;
>  
>  ;; Platforms that QEMU can emulate.
> -(define-record-type <qemu-platform>
> -  (qemu-platform name family magic mask)
> +(define-record-type* <qemu-platform>

Since this is for internal consumption, I’m in favor of keeping plain
‘define-record-type’.  Also, I don’t think the “F” flag belongs here,
it’s mostly orthogonal.

>  (define qemu-binfmt-service-type
>    ;; TODO: Make a separate binfmt_misc service out of this?
>    (service-type (name 'qemu-binfmt)
> @@ -800,9 +833,7 @@ given QEMU package."
>                                            (const
>                                             (list %binary-format-file-system)))
>                         (service-extension shepherd-root-service-type
> -                                          qemu-binfmt-shepherd-services)
> -                       (service-extension guix-service-type
> -                                          qemu-binfmt-guix-chroot)))
> +                                          qemu-binfmt-shepherd-services)))

As discussed on IRC, the downside of this approach is increased disk and
memory footprint (those big binaries have to be loaded in memory).

One possibility would be to add an option to choose between this and the
current approach, but maybe it’s not worth the maintenance trouble.

Thoughts?

Thanks for fixing this issue!

Ludo’.

bug#36117: [PATCH 4/4] services/qemu-binfmt: Use the F flag and the static output of QEMU.

[Maxim Cournoyer]

Hello again,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

>>  ;; Platforms that QEMU can emulate.
>> -(define-record-type <qemu-platform>
>> -  (qemu-platform name family magic mask)
>> +(define-record-type* <qemu-platform>
>
> Since this is for internal consumption, I’m in favor of keeping plain
> ‘define-record-type’.  Also, I don’t think the “F” flag belongs here,
> it’s mostly orthogonal.

Even though it's still internal for now, the intent here was to allow
the advanced users (those who would use (@@ ...) to access what they
need) to configure the qemu-platform objects should they have special
needs for it.

I also don't think the flags belong to a qemu-platform object, but the
same can be said for the magic field.  The most elegant thing would be
to have a binfmt_misc service which we would then extend for QEMU, but
that's more work and the use cases appear to be rare outside of QEMU (I
can only think of WINE).  If you feel strongly about it I can revert
those hunks and hard-coded the F flag.

>>  (define qemu-binfmt-service-type
>>    ;; TODO: Make a separate binfmt_misc service out of this?
>>    (service-type (name 'qemu-binfmt)
>> @@ -800,9 +833,7 @@ given QEMU package."
>>                                            (const
>>                                             (list %binary-format-file-system)))
>>                         (service-extension shepherd-root-service-type
>> -                                          qemu-binfmt-shepherd-services)
>> -                       (service-extension guix-service-type
>> -                                          qemu-binfmt-guix-chroot)))
>> +                                          qemu-binfmt-shepherd-services)))
>
> As discussed on IRC, the downside of this approach is increased disk and
> memory footprint (those big binaries have to be loaded in memory).

The 'big' binaries are not that bigger than what we already have.  For a
typical one:

$ du -h {/gnu/store/7w04gv6m92n40dainn4s6xr3l20r90xw-qemu-5.1.0,\
  /gnu/store/wqh2dyskzkl4vjn6harclyl317h4vfaf-qemu-5.1.0-static}/bin/qemu-arm

4.6M    /gnu/store/7w04gv6m92n40dainn4s6xr3l20r90xw-qemu-5.1.0/bin/qemu-arm
6.4M    /gnu/store/wqh2dyskzkl4vjn6harclyl317h4vfaf-qemu-5.1.0-static/bin/qemu-arm

Only the registered QEMU architectures would be preloaded; so if you
enable 5, the increase it at worst ~ 30 MiB.  Nothing too worrying on
machines that should be quite capable for transparent emulation
purposes, in my opinion.

> One possibility would be to add an option to choose between this and the
> current approach, but maybe it’s not worth the maintenance trouble.

I would rather K.I.S.S., with this only case that works everywhere and
thus doesn't surprise anyone.

> Thanks for fixing this issue!

Thanks for the review!

Maxim

bug#36117: qemu-binfmt with non-native chroot

[Maxim Cournoyer]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hello again,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>>>  ;; Platforms that QEMU can emulate.
>>> -(define-record-type <qemu-platform>
>>> -  (qemu-platform name family magic mask)
>>> +(define-record-type* <qemu-platform>
>>
>> Since this is for internal consumption, I’m in favor of keeping plain
>> ‘define-record-type’.  Also, I don’t think the “F” flag belongs here,
>> it’s mostly orthogonal.
>
> Even though it's still internal for now, the intent here was to allow
> the advanced users (those who would use (@@ ...) to access what they
> need) to configure the qemu-platform objects should they have special
> needs for it.
>
> I also don't think the flags belong to a qemu-platform object, but the
> same can be said for the magic field.  The most elegant thing would be
> to have a binfmt_misc service which we would then extend for QEMU, but
> that's more work and the use cases appear to be rare outside of QEMU (I
> can only think of WINE).  If you feel strongly about it I can revert
> those hunks and hard-coded the F flag.
>
>>>  (define qemu-binfmt-service-type
>>>    ;; TODO: Make a separate binfmt_misc service out of this?
>>>    (service-type (name 'qemu-binfmt)
>>> @@ -800,9 +833,7 @@ given QEMU package."
>>>                                            (const
>>>                                             (list %binary-format-file-system)))
>>>                         (service-extension shepherd-root-service-type
>>> -                                          qemu-binfmt-shepherd-services)
>>> -                       (service-extension guix-service-type
>>> -                                          qemu-binfmt-guix-chroot)))
>>> +                                          qemu-binfmt-shepherd-services)))
>>
>> As discussed on IRC, the downside of this approach is increased disk and
>> memory footprint (those big binaries have to be loaded in memory).
>
> The 'big' binaries are not that bigger than what we already have.  For a
> typical one:
>
> $ du -h {/gnu/store/7w04gv6m92n40dainn4s6xr3l20r90xw-qemu-5.1.0,\
>   /gnu/store/wqh2dyskzkl4vjn6harclyl317h4vfaf-qemu-5.1.0-static}/bin/qemu-arm
>
> 4.6M    /gnu/store/7w04gv6m92n40dainn4s6xr3l20r90xw-qemu-5.1.0/bin/qemu-arm
> 6.4M    /gnu/store/wqh2dyskzkl4vjn6harclyl317h4vfaf-qemu-5.1.0-static/bin/qemu-arm
>
> Only the registered QEMU architectures would be preloaded; so if you
> enable 5, the increase it at worst ~ 30 MiB.  Nothing too worrying on
> machines that should be quite capable for transparent emulation
> purposes, in my opinion.
>
>> One possibility would be to add an option to choose between this and the
>> current approach, but maybe it’s not worth the maintenance trouble.
>
> I would rather K.I.S.S., with this only case that works everywhere and
> thus doesn't surprise anyone.
>
>> Thanks for fixing this issue!
>
> Thanks for the review!
>
> Maxim

I went ahead and pushed this change to master alongside the update to
QEMU 5.2.0.

If there are improvements/changes to be made, we can pick it up from
there!

Closing,

Maxim