bug#61011: ‘guix system vm’ spawns QEMU and hangs

bug#61011: ‘guix system vm’ spawns QEMU and hangs

[Ludovic Courtès]

Hello,

On my Guix System machine, the ‘qemu-system-x86_64’ spawned by ‘guix
system vm’ hangs after printing “Booting from ROM...”; it has to be
terminated with SIGKILL, SIGINT is not enough.

Specifically:

  $(guix time-machine --commit=66188398c446bdf9ce044fa539536e9b54c28c60 \
    -- system vm gnu/system/examples/bare-bones.tmpl) -m 1024  # Good.

… whereas:

  $(guix time-machine --commit=9923100a42ffa80f604c1c13a5e999e6a4c15146 \
    -- system vm gnu/system/examples/bare-bones.tmpl) -m 1024  # Bad!

I thought the culprit might be this commit:

  commit 9923100a42ffa80f604c1c13a5e999e6a4c15146
  Date:   Fri Dec 23 09:42:27 2022 +0200

      gnu: sgabios: Fix build on cross-build architectures.

      * gnu/packages/firmware.scm (sgabios)[arguments]: When cross-building
      add a make-flag to use the correct objcopy.

… but even after reverting it on today’s master, QEMU occasionally hangs
as before, though not always.

‘qemu-minimal’ as used for “make check-system” seems to work fine.

There have been a number of packages unbundled, so I wonder if another
one of these might be causing problems.

What do you think?

Ludo’.

bug#61011: ‘guix system vm’ spawns QEMU and hangs

[Leo Famulari]

On Sun, Jan 22, 2023 at 10:36:21PM +0100, Ludovic Courtès wrote:
>   $(guix time-machine --commit=9923100a42ffa80f604c1c13a5e999e6a4c15146 \
>     -- system vm gnu/system/examples/bare-bones.tmpl) -m 1024  # Bad!

I can reproduce.

It seems to work fine without '-m 1024', in which case it only has 512
MB RAM.

bug#61011: ‘guix system vm’ spawns QEMU and hangs

[Josselin Poiret via Bug reports for GNU Guix]

Hi,
Leo Famulari <leo@famulari.name> writes:

> On Sun, Jan 22, 2023 at 10:36:21PM +0100, Ludovic Courtès wrote:
>>   $(guix time-machine --commit=9923100a42ffa80f604c1c13a5e999e6a4c15146 \
>>     -- system vm gnu/system/examples/bare-bones.tmpl) -m 1024  # Bad!
>
> I can reproduce.
>
> It seems to work fine without '-m 1024', in which case it only has 512
> MB RAM.

This is probably due to the following kernel bug [1], which could be
related to the khugepaged hangs I'm getting on my system since 6.1.

[1]
https://lore.kernel.org/kvm/b8017e09-f336-3035-8344-c549086c2340@kernel.org/ 


Best,
-- 
Josselin Poiret

bug#61011: ‘guix system vm’ spawns QEMU and hangs

[Ludovic Courtès]

Hello,

Josselin Poiret <dev@jpoiret.xyz> skribis:

> Leo Famulari <leo@famulari.name> writes:
>
>> On Sun, Jan 22, 2023 at 10:36:21PM +0100, Ludovic Courtès wrote:
>>>   $(guix time-machine --commit=9923100a42ffa80f604c1c13a5e999e6a4c15146 \
>>>     -- system vm gnu/system/examples/bare-bones.tmpl) -m 1024  # Bad!
>>
>> I can reproduce.
>>
>> It seems to work fine without '-m 1024', in which case it only has 512
>> MB RAM.
>
> This is probably due to the following kernel bug [1], which could be
> related to the khugepaged hangs I'm getting on my system since 6.1.
>
> [1]
> https://lore.kernel.org/kvm/b8017e09-f336-3035-8344-c549086c2340@kernel.org/ 

Ouch.  I’m running 6.1 since January 16th, which is about the time I
first experienced the issue.

Ludo’.

bug#61011: ‘guix system vm’ spawns QEMU and hangs

[Efraim Flashner]

[-- Attachment #1.1: Type: text/plain, Size: 2294 bytes --]

On Sun, Jan 22, 2023 at 10:36:21PM +0100, Ludovic Courtès wrote:
> Hello,
> 
> On my Guix System machine, the ‘qemu-system-x86_64’ spawned by ‘guix
> system vm’ hangs after printing “Booting from ROM...”; it has to be
> terminated with SIGKILL, SIGINT is not enough.
> 
> Specifically:
> 
>   $(guix time-machine --commit=66188398c446bdf9ce044fa539536e9b54c28c60 \
>     -- system vm gnu/system/examples/bare-bones.tmpl) -m 1024  # Good.
> 
> … whereas:
> 
>   $(guix time-machine --commit=9923100a42ffa80f604c1c13a5e999e6a4c15146 \
>     -- system vm gnu/system/examples/bare-bones.tmpl) -m 1024  # Bad!
> 
> I thought the culprit might be this commit:
> 
>   commit 9923100a42ffa80f604c1c13a5e999e6a4c15146
>   Date:   Fri Dec 23 09:42:27 2022 +0200
> 
>       gnu: sgabios: Fix build on cross-build architectures.
> 
>       * gnu/packages/firmware.scm (sgabios)[arguments]: When cross-building
>       add a make-flag to use the correct objcopy.
> 
> … but even after reverting it on today’s master, QEMU occasionally hangs
> as before, though not always.
> 
> ‘qemu-minimal’ as used for “make check-system” seems to work fine.
> 
> There have been a number of packages unbundled, so I wonder if another
> one of these might be causing problems.
> 
> What do you think?

I remember feeling overwhelmed by the build failures after the
unbundling (but I didn't reach out! I should've said something.) and
worked to try and quickly fix the builds.

I looked at reverting it locally, but with or without that patch I got
the same derivation for sgabios when built on x86_64. I tried firing up
diffoscope and I found no differences between the sgabios built on
x86_64, aarch64 or armhf (wow!).

I've run diffoscope against the sga I tried firing up diffoscope and I
found no differences between the sgabios built on x86_64, aarch64 or
armhf (wow!).

I've run diffoscope against the sgabios.bin that we build and the one
that comes in the qemu release tarball and I've included the output in
the email.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: sgabios.bin.diff.gz --]
[-- Type: application/x-gunzip, Size: 8092 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]