bug#63197: video acceleration/libva segfaults caused by stale mesa shader cache

bug#63197: video acceleration/libva segfaults caused by stale mesa shader cache

[Maxim Cournoyer]

Hi,

After reinstalling someone's desktop which has support for VA-API,
'vainfo' from 'libva-utils' would consume all the memory then crash.
Other applications relying on libva would crash as well, e.g. ffmpeg (or
its users, such as vlc/jami).  Here's a sample output from VLC:

--8<---------------cut here---------------start------------->8---
vlc received_605209834855384.mp4 
VLC media player 3.0.18 Vetinari (revision 3.0.13-8-g41878ff4f2)
[000000000109d770] main libvlc: Lancement de vlc avec l'interface par défaut. Utiliser « cvlc » pour démarrer VLC sans interface.
libva info: VA-API version 1.17.0
libva info: Trying to open /gnu/store/9pypr3c3y379shbwm9ilb4pik9mkfd83-mesa-22.2.4/lib/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_17
Erreur de segmentation
--8<---------------cut here---------------end--------------->8---

After tracing the process, I noticed that the last thing it did was
loading its mesa shader cache, stored under:

--8<---------------cut here---------------start------------->8---
~/.cache/mesa_shader_cache
--8<---------------cut here---------------end--------------->8---

Deleting that directory resolved the issue.

It seems that'd be a bug in Mesa (for failing to determine that it
should have invalidated its cache going from version 21 to 22 post
core-updates merge).

-- 
Thanks,
Maxim

bug#63197: video acceleration/libva segfaults caused by stale mesa shader cache

[Maxim Cournoyer]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> After reinstalling someone's desktop which has support for VA-API,
> 'vainfo' from 'libva-utils' would consume all the memory then crash.
> Other applications relying on libva would crash as well, e.g. ffmpeg (or
> its users, such as vlc/jami).  Here's a sample output from VLC:
>
> vlc received_605209834855384.mp4 
> VLC media player 3.0.18 Vetinari (revision 3.0.13-8-g41878ff4f2)
> [000000000109d770] main libvlc: Lancement de vlc avec l'interface par défaut. Utiliser « cvlc » pour démarrer VLC sans interface.
> libva info: VA-API version 1.17.0
> libva info: Trying to open /gnu/store/9pypr3c3y379shbwm9ilb4pik9mkfd83-mesa-22.2.4/lib/dri/radeonsi_drv_video.so
> libva info: Found init function __vaDriverInit_1_17
> Erreur de segmentation
>
>
> After tracing the process, I noticed that the last thing it did was
> loading its mesa shader cache, stored under:
>
> ~/.cache/mesa_shader_cache
>
> Deleting that directory resolved the issue.
>
> It seems that'd be a bug in Mesa (for failing to determine that it
> should have invalidated its cache going from version 21 to 22 post
> core-updates merge).

I've forwarded this report upstream here:
https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937

-- 
Thanks,
Maxim

bug#63197: video acceleration/libva segfaults caused by stale mesa shader cache

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

Hi Maxim,

I learned about this issue today

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

>> After tracing the process, I noticed that the last thing it did was
>> loading its mesa shader cache, stored under:
>>
>> ~/.cache/mesa_shader_cache
>>
>> Deleting that directory resolved the issue.
>>
>> It seems that'd be a bug in Mesa (for failing to determine that it
>> should have invalidated its cache going from version 21 to 22 post
>> core-updates merge).

AFAIU this issue is still present using mesa 23 since Guillaume Le
Vaillant had to use this workaround yesterday [1] and reported his
backtrace upstream [2]

If I'm not wrong (i.e. vlc et al are now using mesa 23) this should also
be reported upstream (I can do it if needed).

AFAIU the only thing we can do to fix this bug is to disable the shader
cache (MESA_SHADER_CACHE_DISABLE=true) until a proper fix is found
upstream.

...or apply a patch to rename "~/.cache/mesa_shader_cache" to
"~/.cache/mesa<version>_shader_cache"

Alternatively, we should find a way to make Guix users aware of this
kind of problems and possible workarounds they can apply (it's not
related to this specific bug)


WDYT?

Thanks! Gio'


[1] id:871qify1i8.fsf@kitej

[2] https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937#note_1960628

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

bug#63197: video acceleration/libva segfaults caused by stale mesa shader cache

[Maxim Cournoyer]

Hello,

Giovanni Biscuolo <g@xelera.eu> writes:

> Hi Maxim,
>
> I learned about this issue today
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>>> After tracing the process, I noticed that the last thing it did was
>>> loading its mesa shader cache, stored under:
>>>
>>> ~/.cache/mesa_shader_cache
>>>
>>> Deleting that directory resolved the issue.
>>>
>>> It seems that'd be a bug in Mesa (for failing to determine that it
>>> should have invalidated its cache going from version 21 to 22 post
>>> core-updates merge).
>
> AFAIU this issue is still present using mesa 23 since Guillaume Le
> Vaillant had to use this workaround yesterday [1] and reported his
> backtrace upstream [2]
>
> If I'm not wrong (i.e. vlc et al are now using mesa 23) this should also
> be reported upstream (I can do it if needed).

Which upstream are you thinking about?  My understanding is that this
problem is a Mesa problem, and it's already reported there (the issue
linked in [2]).

> AFAIU the only thing we can do to fix this bug is to disable the shader
> cache (MESA_SHADER_CACHE_DISABLE=true) until a proper fix is found
> upstream.

Disabling the shader cache sounds like a decent workaround or even
definitive solution.  One less stale cache to worry about...  If it's
like the Qt shader cache, the performance hit is probably too small to
be noticeable (maybe just slower startup times of complicated opengl
applications such as games?).

> ...or apply a patch to rename "~/.cache/mesa_shader_cache" to
> "~/.cache/mesa<version>_shader_cache"

That's another good idea.

> Alternatively, we should find a way to make Guix users aware of this
> kind of problems and possible workarounds they can apply (it's not
> related to this specific bug)

I would rather pursue the other above options you suggest, so that it
doesn't happen in the first place!

Thank you for sharing these ideas.

-- 
Maxim

bug#63197: video acceleration/libva segfaults caused by stale mesa shader cache

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 3415 bytes --]

Hi Maxim

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

>> AFAIU this issue is still present using mesa 23 since Guillaume Le
>> Vaillant had to use this workaround yesterday [1] and reported his
>> backtrace upstream [2]
>>
>> If I'm not wrong (i.e. vlc et al are now using mesa 23) this should also
>> be reported upstream (I can do it if needed).
>
> Which upstream are you thinking about?

mesa

> My understanding is that this problem is a Mesa problem, and it's
> already reported there (the issue linked in [2]).

yes but the original bug report mentions Mesa 22.2.4 and M. Briar asked:

--8<---------------cut here---------------start------------->8---

Mesa 22.2.x is already end-of-life and won't receive any fixes
anymore. Does this also happen on newer versions?

--8<---------------cut here---------------end--------------->8---
(https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937#note_1891435)

IMHO there is no clear answer to that question in the bug thread, maybe
mesa developers still think it's just 22.2.X related

Now we have Mesa 23.0.3 in Giux, probably the one used by vlc when
Guillaume reported his issue upstream (mesa) on June 15

>> AFAIU the only thing we can do to fix this bug is to disable the shader
>> cache (MESA_SHADER_CACHE_DISABLE=true) until a proper fix is found
>> upstream.
>
> Disabling the shader cache sounds like a decent workaround or even
> definitive solution.  One less stale cache to worry about...

oh yes!  Unfortunately cache management is not so robust... sometimes :-(

> If it's like the Qt shader cache, the performance hit is probably too
> small to be noticeable (maybe just slower startup times of complicated
> opengl applications such as games?).

I don't know the cost in term of performance, I'm not a 3D expert at
all; from what I read on the web about shader chaches I guess it's a
real problem almost only for games and I guess it's not a problem at
all for media players like vlc et al: I'm just brainstorming but what
about having a mesa-with-cache-enabled version just for the games, if it
is really needed?

I should be able to propose a patch to disable the mesa shader cache,
but since I'm not an expert in this field I prefer to leave this
decision (to disable the cache, I mean) to someone else

>> ...or apply a patch to rename "~/.cache/mesa_shader_cache" to
>> "~/.cache/mesa<version>_shader_cache"
>
> That's another good idea.

I was just doing guesswork but the bug caused by this mesa upgrade
smells like a binary incompatibility between two versions (or just major
versions)... so a versioned shader cache makes sense to me

I'm not able to propose (I mean to code) such a patch, anyway

Anyway, users should know that they have to periodically clean unused
shader caches, since from what I read on the net the shader cache tends
to really /explode/ in terms of size, in some cases

>> Alternatively, we should find a way to make Guix users aware of this
>> kind of problems and possible workarounds they can apply (it's not
>> related to this specific bug)
>
> I would rather pursue the other above options you suggest, so that it
> doesn't happen in the first place!

I agree

> Thank you for sharing these ideas.

Thank you for your attention!

Happy hacking, Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

bug#63197: video acceleration/libva segfaults caused by stale mesa shader cache

[Maxim Cournoyer]

Hi Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:

[...]

>>> ...or apply a patch to rename "~/.cache/mesa_shader_cache" to
>>> "~/.cache/mesa<version>_shader_cache"
>>
>> That's another good idea.
>
> I was just doing guesswork but the bug caused by this mesa upgrade
> smells like a binary incompatibility between two versions (or just major
> versions)... so a versioned shader cache makes sense to me
>
> I'm not able to propose (I mean to code) such a patch, anyway
>
> Anyway, users should know that they have to periodically clean unused
> shader caches, since from what I read on the net the shader cache tends
> to really /explode/ in terms of size, in some cases
>
>>> Alternatively, we should find a way to make Guix users aware of this
>>> kind of problems and possible workarounds they can apply (it's not
>>> related to this specific bug)
>>
>> I would rather pursue the other above options you suggest, so that it
>> doesn't happen in the first place!

I've ping'd upstream with
https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937#note_1975560.
Let's see what they say!

-- 
Thanks,
Maxim

bug#63197: video acceleration/libva segfaults caused by stale mesa shader cache

[Maxim Cournoyer]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> After reinstalling someone's desktop which has support for VA-API,
> 'vainfo' from 'libva-utils' would consume all the memory then crash.
> Other applications relying on libva would crash as well, e.g. ffmpeg (or
> its users, such as vlc/jami).  Here's a sample output from VLC:
>
> vlc received_605209834855384.mp4 
> VLC media player 3.0.18 Vetinari (revision 3.0.13-8-g41878ff4f2)
> [000000000109d770] main libvlc: Lancement de vlc avec l'interface par défaut. Utiliser « cvlc » pour démarrer VLC sans interface.
> libva info: VA-API version 1.17.0
> libva info: Trying to open /gnu/store/9pypr3c3y379shbwm9ilb4pik9mkfd83-mesa-22.2.4/lib/dri/radeonsi_drv_video.so
> libva info: Found init function __vaDriverInit_1_17
> Erreur de segmentation

The same issue was reproduced with vlc and Totem after the latest
upgrades, which brought mesa from version 23 to 24.  The same solution:

--8<---------------cut here---------------start------------->8---
$ rm -rf ./.cache/mesa_shader_cache
--8<---------------cut here---------------end--------------->8---

still works.

I've sent upstream a fresh snapshot of the 23 -> 24 corrupted cache [0].

[0]  https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937#note_2388471

-- 
Thanks,
Maxim