X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

Hi,

My last system reconfigure dates back to April 17th and works
flawlessly.
Yesterday I ran a new one and X / GDM don't start up properly:

- I can guess bits and pieces of the GDM interface but the screen is
  torn asunder by the most epic artifacts (green / purple flashing tiles).

- In a TTY, I see that GDM is restarting in an infinite loop.

- dmesg reports some "corruption" at the level of the video card (AMD
  Radeon RX 580).

The weird thing is that I'm still using Linux 5.0.7 like I was on April
17th and everything worked well.

I've tried both with linux-libre and a custom kernel.

Have we changed anything to the boot parameters maybe?


Side question: how do we know which Guix commit was used for a specific
system generation?  For now the only way I can think of is

--8<---------------cut here---------------start------------->8---
/var/guix/profiles/system-17-link/profile/bin/guix --version
--8<---------------cut here---------------end--------------->8---

Anything more convenient?  From Emacs Guix?


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[ison]

I can confirm this exact same issue with the same video card, RX 580, and my
last attempt to update pulled in kernel 5.0.13. When the login screen shows up
the screen is horribly corrupt, nothing is even recognizable as being a part of
the normal login screen, just random colored lines all over the screen.
I first experienced the issue at the release of Guix v1.0.0, but wasn't sure if
I should wait before reporting since I couldn't be sure if it was a guix
specific issue or something else.

Here is what I have discovered:
* I doubt the issue is either caused by GDM itself or the kernel, because after
  updating from a working generation (but without restarting my machine) if I
  attempt to play a video in mpv (or start a game) the same graphical artifacts
  appear, but only in that 1 window.
* I have attempted to upgrade individual applications while using
  "package-input-rewriting" to replace their dependencies on Mesa 18.3.5 with
  18.3.1 (the version they're using on my working generation) and I can confirm
  that the replacement succeeded with "ldd", however they still receive
  graphical artifacts.
* I have successfully managed to run an updated copy of mpv without artifacts by
  using LD_PRELOAD to force it to point to the exact copy of libGL.so.1 used by
  mpv on my working generation.

This leads me to believe that the problem lies somewhere in Mesa (which contains
libGL.so.1). However, the issue doesn't seem to be just the version of Mesa
since mpv still had corrupt video when built to use the older version of Mesa,
but had no corrupt video when forced to link to the exact copy of libGL.so.1
from a previously working generation.

Note: mpv was merely used as a test so I didn't have to constantly restart my
machine. However, generations which resulted in corrupt video in mpv (and games)
always resulted in corrupt video in GDM if I also rebooted into that generation.

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 75 bytes --]

Might be related to LLVM because it's sometimes used by shader compilers.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[ison]

I think you're right. I just tried different versions of llvm with the game
minetest (testing with minetest now since it compiles much faster than mpv)

Trying with llvm@7.0.1 using the command:
guix environment --ad-hoc minetest --with-input=llvm=llvm@7.0.1
results in the same corrupt video issue

Trying with llvm@6.0.1 using the command:
guix environment --ad-hoc minetest --with-input=llvm=llvm@6.0.1
results in no corruption and a fully functioning game

Thanks for helping to narrow down the issue. Although I'm not sure where to go
from here.

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

On Thu, 9 May 2019 19:59:17 -0600
ison <ison@airmail.cc> wrote:

> Thanks for helping to narrow down the issue. Although I'm not sure where to go
> from here.

Maybe try mesa 18.3.6, maybe someone fixed it there already.
Apparently there's also mesa 19.0.4.
If it can't be fixed otherwise, we can compile the old mesa with llvm 6.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[ison]

Sorry I neglected to mention that I actually tried upgrading mesa earlier and it
didn't resolve the problem. Neither upgrading mesa (I tried 18.3.6 and some 19.*
version) nor downgrading (back to 18.3.1) would fix it, all rebuilds still
resulted in artifacts. I suppose it's because the dependency on llvm 7 remained
fixed during all the rebuilds.

I also just tried llvm 8.0.0 and it still had the same issue.

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

Thanks for investigating this!

So does anyone know if there is a good reason to keep llvm 7 for Mesa?
Was it just updated for the sake of updating?  (Which is usually a good
reason, admittedly :p)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

ison <ison@airmail.cc> writes:

> Sorry I neglected to mention that I actually tried upgrading mesa earlier and it
> didn't resolve the problem. Neither upgrading mesa (I tried 18.3.6 and some 19.*
> version) nor downgrading (back to 18.3.1) would fix it, all rebuilds still
> resulted in artifacts. I suppose it's because the dependency on llvm 7 remained
> fixed during all the rebuilds.
>
> I also just tried llvm 8.0.0 and it still had the same issue.

Can you try to delete "$HOME/.cache/mesa_shader_cache/" and see if see
if it makes a difference?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Boris A. Dekshteyn]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Hi,
>
> My last system reconfigure dates back to April 17th and works
> flawlessly.
> Yesterday I ran a new one and X / GDM don't start up properly:
>
> - I can guess bits and pieces of the GDM interface but the screen is
>   torn asunder by the most epic artifacts (green / purple flashing tiles).
>
> - In a TTY, I see that GDM is restarting in an infinite loop.
>
> - dmesg reports some "corruption" at the level of the video card (AMD
>   Radeon RX 580).
>
> The weird thing is that I'm still using Linux 5.0.7 like I was on April
> 17th and everything worked well.
>
> I've tried both with linux-libre and a custom kernel.
>
> Have we changed anything to the boot parameters maybe?
>
>
> Side question: how do we know which Guix commit was used for a specific
> system generation?  For now the only way I can think of is
>
> /var/guix/profiles/system-17-link/profile/bin/guix --version
>
> Anything more convenient?  From Emacs Guix?

See bug#35575. If you have the same symptoms, the problem is most likely
that mesa does not check its shader cache. Working solution - clear
$HOME/.cache/mesa_shader_cache/

-- 
WBR, Boris Dekshteyn

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 117 bytes --]

Hmm, not sure it's related to user data since GDM fails to display.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 177 bytes --]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Hmm, not sure it's related to user data since GDM fails to display.

Also try to `rm -rf /var/lib/gdm/.cache/mesa_shader_cache`.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 108 bytes --]

OK, thanks for the tip!  Will do when I'm back on Tuesday.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[ison]

On Fri, May 10, 2019 at 02:11:48PM +0200, Marius Bakke wrote:
> Can you try to delete "$HOME/.cache/mesa_shader_cache/" and see if see
> if it makes a difference?

That did it! At least for the user installed applications I was testing with it seems to have resolved the issue.
I also deleted the other cache you mentioned, at: /var/lib/gdm/.cache/mesa_shader_cache
and will now try to do a full system upgrade and see if everything is working.

Thanks for the help.

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[ison]

Ok this solution has resolved the issue for me on a newly updated system.

Note that deleting both shader caches was required, and also if the caches get
rebuilt on a new generation and then I try to boot into an older previously
working generation then that generation will display graphics artifacts until
the caches are deleted again.

So switching between mesa compiled with llvm 6 and 7 on AMD RX 580 either
backward or forward requires manually deleting the shader caches.

Thanks again for the help.

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

Hello Guix,

ison <ison@airmail.cc> writes:

> Ok this solution has resolved the issue for me on a newly updated system.
>
> Note that deleting both shader caches was required, and also if the caches get
> rebuilt on a new generation and then I try to boot into an older previously
> working generation then that generation will display graphics artifacts until
> the caches are deleted again.

since cache status issues are a common pattern (AFAIU) I've filed a bug
report titled: "wishlist: addessing statefulness of .cache(s)"
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35683

happy hacking! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 301 bytes --]

Giovanni Biscuolo <g@xelera.eu> writes:

> since cache status issues are a common pattern (AFAIU) I've filed a bug
> report titled: "wishlist: addessing statefulness of .cache(s)"
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35683

Good call!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 119 bytes --]

Tried removing both shader caches and it worked!  Thanks for the tip!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 276 bytes --]

I'm still having issues: whenever I use some applications like ioquake3
(a 3D OpenGL2 game engine) then mpv and chromium show crazy rendering
artifacts.

I have to clear the cache after every call to ioquake3.

Any idea?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[ison]

Are you using GuixSD or Guix on top of another distro?

It seemed to me that when my caches were empty the first application I'd run
would cause the shaders to be rebuilt with whatever version of llvm that
application was built with.
And it also seems that the issue goes both ways, so if you run an application
that builds the shaders with llvm6 then all applications which expect 7 will
show artifacts, and vice versa, if you clear your cache and then immediately run
a program which builds the shaders with llvm7 then any programs which expect
llvm6 will see artifacts.

So it sounds to me like you have a combination of programs built with these
conflicting versions.
In my case, since I'm running GuixSD, I just did a full upgrade, cleared both
caches, and then restarted my machine. Haven't had an issue since.

However, if you're using just Guix on another distro, then perhaps the host
distro software uses llvm6 while your Guix applications use llvm7?
If that's the case then I'm not sure what the best solution to that would be.
Perhaps you could just consider re-installing the affected Guix applications
with:
guix package -i your_program --with-input=llvm=llvm@6.0.1
until your host distro updates to llvm7.

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 267 bytes --]

I'm using Guix System.

The different versions of llvm were a good hunch, unfortunately the
offending programs (ungoogled-chromium and ioquake3) use the same
/gnu/store/slhclazbx16ij6kyggz667sycmyy9iii-llvm-7.0.1.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: X/GDM fails on linux 5.0.7 AMD Radeon RX 580 after system reconfigure

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 209 bytes --]

After a `guix package -m manifest.scm`, everything seems to be fine.
I guess one of the programs I was using was still depending on llvm 6.

Problem solved!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]