bug#53712: Guix System hangs after boot with linux-libre 5.15.17

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

Guix System on x86_64 hangs after boot when reconfigured to use
linux-libre 5.15.17.

The system becomes unresponsive to keyboard input after boot. Sometimes
it reaches the login prompt, sometimes not. Services that take a while
to start, typically finishing after the prompt is displayed, do not
ever start.

I bisected and confirmed that the problem is introduced in Guix Git
commit aad96ed54070 "gnu: linux-libre: Update to 5.15.17."

I could not reproduce the problem with `guix system vm [...]`. Maybe
this means that the problem only occurs after reconfiguring, rather than
a fresh system, or maybe it just doesn't occur at all in a KVM VM.

I'll try with 5.15.18 and the 5.10 series. We are still unable to deploy
linux-libre 5.16 for Guix System users due to this bug:

https://issues.guix.gnu.org/53554

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Maxim Cournoyer]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> Guix System on x86_64 hangs after boot when reconfigured to use
> linux-libre 5.15.17.
>
> The system becomes unresponsive to keyboard input after boot. Sometimes
> it reaches the login prompt, sometimes not. Services that take a while
> to start, typically finishing after the prompt is displayed, do not
> ever start.
>
> I bisected and confirmed that the problem is introduced in Guix Git
> commit aad96ed54070 "gnu: linux-libre: Update to 5.15.17."
>
> I could not reproduce the problem with `guix system vm [...]`. Maybe
> this means that the problem only occurs after reconfiguring, rather than
> a fresh system, or maybe it just doesn't occur at all in a KVM VM.
>
> I'll try with 5.15.18 and the 5.10 series. We are still unable to deploy
> linux-libre 5.16 for Guix System users due to this bug:

Not sure if that helps, but my system is running smoothly on 5.15.18 as
I write this.  What are the specifics of your system?

Thanks,

Maxim

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

On Tue, Feb 01, 2022 at 04:11:27PM -0500, Maxim Cournoyer wrote:
> Not sure if that helps, but my system is running smoothly on 5.15.18 as
> I write this.  What are the specifics of your system?

Interestingly, for me 5.15.18 does boot to an interactive console, but
the system fails to bring up the wifi interface, and then it fails to
halt upon command, just hanging forever. At least, it did that for the 3
times I tried using that generation.

This is a Thinkpad x200s with 3 GB RAM. Normal BIOS. I've attached the
operating-system declaration.

There are similar reports on #guix IRC from two other users:

http://logs.guix.gnu.org/guix/2022-01-31.log#152606
http://logs.guix.gnu.org/guix/2022-01-30.log#005034

[-- Attachment #2: system.scm --]
[-- Type: text/plain, Size: 3793 bytes --]

;; This is an operating system configuration template
;; for a "bare bones" setup, with no X11 display server.

(use-modules (gnu))
(use-service-modules networking
                     desktop
                     dbus
                     ssh)
(use-package-modules admin
                     certs
                     curl
                     linux
                     ntp
                     nvi
                     ssh
                     rsync
                     tmux
                     version-control
                     wicd
                     vim)

(operating-system
  (host-name "zamia")
  (timezone "America/New_York")
  (locale "en_US.UTF-8")

  (kernel-loadable-modules (list rtl8812au-aircrack-ng-linux-module))
  (kernel-arguments
   '(;; Console resolution
     "gfxpayload=1440x900x16,1440x900"

     ;; console cursor. stops the blinking but the colors are bad
     "vt.cur.default=0x520032"

     "consoleblank=120"
     ;; ???
     "quiet"
     ;; Disable the PC speaker
     "modprobe.blacklist=pcspkr,snd_pcsp"))

  ;; Assuming /dev/sdX is the target hard disk, and "my-root" is
  ;; the label of the target root file system.
  (bootloader (grub-configuration (target "/dev/sda")
                                  (terminal-outputs '(console))))
  (file-systems (cons* (file-system
                        (device (uuid "0pa2dcd8-e037-43fb-b0cc-9ec5bcc3127a"))
                        (mount-point "/")
                        (type "btrfs")
                        (options "compress-force=zstd"))
                       (file-system
                        (device (uuid "9p614cc2-af95-482a-b906-ebc958ed57b7"))
                        (mount-point "/home")
                        (type "btrfs")
                        (options "compress-force=zstd"))

		       ; This will break the boot
		       ; <https://bugs.gnu.org/35472>
;		       (file-system
;			 (device "/foo/bar")
;			 (mount-point "/bar")
;			 (type "none")
;			 (check? #f)
;			 (needed-for-boot? #t)
;			 (flags '(bind-mount)))
                      %base-file-systems))

  ;; This is where user accounts are specified.  The "root"
  ;; account is implicit, and is initially created with the
  ;; empty password.
  (users (append (list (user-account
                         (name "leo")
                         (group "users")
                         ;; Adding the account to the "wheel" group
                         ;; makes it a sudoer.  Adding it to "audio"
                         ;; and "video" allows the user to play sound
                         ;; and access the webcam.
                         (supplementary-groups '("wheel" "netdev" "audio"))))
               %base-user-accounts))

  ;; Globally-installed packages.
  (packages (append (list curl
                          atop htop
                          git
                          openssh mosh
                          nss-certs
                          ntp
                          rsync
                          tmux
                          tree
                          vim nvi)
                    %base-packages-disk-utilities
                    %base-packages))

  (services
    (append
      (list (dbus-service)
            (service gpm-service-type)
            (service openssh-service-type
                       (openssh-configuration
                         (password-authentication? #f)))
            (service ntp-service-type)
	    (service wicd-service-type wicd)
            (elogind-service))
       (modify-services %base-services
         (guix-service-type config =>
                            (guix-configuration
                              (inherit config)
                              (substitute-urls
                                '("https://custom.example.com"))))))))

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

On Tue, Feb 01, 2022 at 03:39:27PM -0500, Leo Famulari wrote:
> I'll try with 5.15.18 and the 5.10 series. We are still unable to deploy
> linux-libre 5.16 for Guix System users due to this bug:

Same problem with 5.10.95.

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

On Tue, Feb 01, 2022 at 03:39:27PM -0500, Leo Famulari wrote:
> Guix System on x86_64 hangs after boot when reconfigured to use
> linux-libre 5.15.17.

More anecdata:

While I build Linux 5.15.17 (not linux-libre) using the Guix kernel
config, adapted for Debian, it works fine on Debian.

The only changes required for Debian are:

1) Switch from CONFIG_MODULE_COMPRESS_GZIP to CONFIG_MODULE_COMPRESS_XZ
2) Set CONFIG_MODPROBE_PATH=/sbin/modprobe

This Debian kernel doesn't use the config options set in ((gnu packages
linux) %default-extra-linux-options), so it's not a very useful point of
comparison.

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

On Tue, Feb 01, 2022 at 03:39:27PM -0500, Leo Famulari wrote:
> Guix System on x86_64 hangs after boot when reconfigured to use
> linux-libre 5.15.17.

More anecdata:

While I build Linux 5.15.17 (not linux-libre) using the Guix kernel
config, adapted for Debian, it works fine on Debian.

The only changes required for Debian are:

1) Switch from CONFIG_MODULE_COMPRESS_GZIP to CONFIG_MODULE_COMPRESS_XZ
2) Set CONFIG_MODPROBE_PATH=/sbin/modprobe

This Debian kernel doesn't use the config options set in ((gnu packages
linux) %default-extra-linux-options), so it's not a very useful point of
comparison.

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

Discussion of the problem on a reddit page about Fedora:

https://old.reddit.com/r/Fedora/comments/shhj6e/kernels_above_51516_are_crashing/

They say that 5.15.18 works for them.

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

The failure exists for me with the most recent versions of the kernel
series 5.4, 5.10, 5.15, and 5.16.

Sometimes the screen does display the end of a kernel panic, but I can
only see the last few lines.

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

I'm still experiencing this problem with 5.15.21, as well 5.16.7 and
5.4.177.

bug#53712: I'm also experiencing this

[Katherine Cox-Buday]

I thought I'd chime in with another data-point:

- Lenovo ThinkPad T480s
- Normal BIOS
- Vanilla Linux kernel(s)
- kernel v5.15.16 boots fine; everything after seems to hit this bug
- Turning off bluetooth in BIOS seemed to get boot further
- Turning off wifi in BIOS allows me to boot

The spec sheet tells me it has a Intel Dual Band Wireless-AC 8265 card, so I guess it's something to do with that. I don't know if this is an upstream kernel bug or something to do with distros specifically?

I hope this helps narrow things down. Luckily my laptop is just used as a desktop, and I have ethernet, so I can just keep Wifi disabled.

-- 
Katherine

bug#53712: I'm also experiencing this

[Leo Famulari]

On Mon, Feb 07, 2022 at 10:54:53AM -0600, Katherine Cox-Buday wrote:
> I thought I'd chime in with another data-point:
> 
> - Lenovo ThinkPad T480s
> - Normal BIOS
> - Vanilla Linux kernel(s)
> - kernel v5.15.16 boots fine; everything after seems to hit this bug
> - Turning off bluetooth in BIOS seemed to get boot further
> - Turning off wifi in BIOS allows me to boot
> 
> The spec sheet tells me it has a Intel Dual Band Wireless-AC 8265 card, so I guess it's something to do with that. I don't know if this is an upstream kernel bug or something to do with distros specifically?
> 
> I hope this helps narrow things down. Luckily my laptop is just used as a desktop, and I have ethernet, so I can just keep Wifi disabled.

Thanks for the notes.

I'm using wifi supported by the drivers from Guix's
rtl8812au-aircrack-ng-linux-module with linux-libre, so it also affects
users who only use the GNU Guix channel.

I also noticed some improvement when removing the wifi dongle, although
I was still unable to properly halt or reboot.

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Squirrel via Bug reports for GNU Guix]

It seems that this is a bug from upstream linux, which may be patched 
soon. Please see 
https://lore.kernel.org/stable/164448100914.10463.9523338503936670263.kvalo@kernel.org/

Blacklisting the iwlwifi kernel module works as a workaround for now, as 
is suggested by Jason Self, the maintainer of the Freesh and libeRTy apt 
repositories of linux-libre kernel. He mentioned it in Trisquel forum at 
https://trisquel.info/en/forum/trisquel-9-linux-libre-51517-will-not-allow-login 
and at https://lore.kernel.org/all/20220203161959.3edf1d6e@valencia/

I'd removed the Intel AX201NGW Wi-Fi & Bluetooth card from my Acer Swift 
3x laptop two days ago. There isn't any wireless network card installed 
now. The system runs without errors, which may be because the iwlwifi 
kernel module is not loaded due to the lack of a Wi-Fi network card. 
Before that, the system hanged randomly with the most recent versions of 
the kernel series 5.4, 5.10 and 5.15.

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

On Fri, Feb 11, 2022 at 04:03:03AM +0800, Squirrel via Bug reports for GNU Guix wrote:
> Blacklisting the iwlwifi kernel module works as a workaround for now, as is
> suggested by Jason Self, the maintainer of the Freesh and libeRTy apt
> repositories of linux-libre kernel. He mentioned it in Trisquel forum at https://trisquel.info/en/forum/trisquel-9-linux-libre-51517-will-not-allow-login
> and at https://lore.kernel.org/all/20220203161959.3edf1d6e@valencia/

Thanks for the info!

I can confirm that adding the modprobe incantation to my
operating-system declaration, reconfiguring, and rebooting does fix the
bug:

(kernel-arguments
 '(; https://issues.guix.gnu.org/53712#12
   "modprobe.blacklist=pcspkr,snd_pcsp,iwlwifi"))

https://guix.gnu.org/manual/devel/en/html_node/operating_002dsystem-Reference.html#FOOT24

bug#53712: Guix System hangs after boot with linux-libre 5.15.17

[Leo Famulari]

On Fri, Feb 11, 2022 at 04:03:03AM +0800, Squirrel via Bug reports for GNU Guix wrote:
> It seems that this is a bug from upstream linux, which may be patched soon.
> Please see https://lore.kernel.org/stable/164448100914.10463.9523338503936670263.kvalo@kernel.org/

Good news, this bug has been fixed in the latest group of stable Linux
releases:

5.16.11
5.15.25
5.10.102
5.4.181
4.19.231
4.14.268
4.9.303

For our default release series, 5.15, it was fixed as commit
ddd46059f7d99119b62d44c519df7a79f2e6a515:

https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.15.25