Re: stateful caches (was Re: OBS Studio memory leak)

[Guillaume Le Vaillant <glv@posteo.net>]

[-- Attachment #1: Type: text/plain, Size: 8608 bytes --]

Giovanni Biscuolo <g@xelera.eu> skribis:

> Hi Guillaume Le Vaillant and Guix Devels,
>
> sorry for cross-posting but IMHO the workaround you found [1] for the memory
> leak affecting a number of media processing applications is of interest
> for many people potentially not subscribed to help-guix
>
> AFAIK this was not filed as a Guix bug
>
> Guillaume Le Vaillant <glv@posteo.net> writes:
>
>> Ott Joon <ott.joon@tutanota.com> skribis:
>>
>>> Hey
>>>
>>> Tried the same thing in VLC and it freezes on GPU accel and starts
>>> leaking memory while also becoming hard to kill.  Maybe this also
>>> explains why some mpv GPU accel settings don't work also in the exact
>>> same way.  I have an AMD RX 6900 XT on this machine.
>
> [...]
>
>> It looks like an issue with the shader cache of mesa.
>> After clearing it, I don't see the memory leak anymore.
>
> good catch: please can you tell us how you managed to spot that problem?
> Did you straced it or did yoy find a related mesa bug report?

I used gdb on versions of mesa and vlc with debug symbols:

--8<---------------cut here---------------start------------->8---
guix build --with-debug-info=mesa --with-debug-info=vlc vlc

gdb /gnu/store/...-vlc-3.0.18/bin/.vlc-real
(gdb) run some-video.mkv
--8<---------------cut here---------------end--------------->8---

Then I sent a SIGSTOP signal to the vlc process, and in gdb I looked at
the backtrace of all the threads of vlc. There was only one thread
allocating things:

--8<---------------cut here---------------start------------->8---
Thread 34

#0  0x00007ffff7e127c7 in sysmalloc () from /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6
#1  0x00007ffff7e13cfb in _int_malloc () from /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6
#2  0x00007ffff7e145b5 in malloc () from /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6
#3  0x00007fff810822b1 in ralloc_size () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#4  0x00007fff81710b2c in read_constant () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#5  0x00007fff817110f2 in read_var_list () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#6  0x00007fff81711865 in nir_deserialize () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#7  0x00007fff8162c874 in tgsi_to_nir () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#8  0x00007fff8146cba1 in si_create_compute_state () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#9  0x00007fff8118e1ba in vl_compositor_cs_create_shader () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#10 0x00007fff8118e800 in vl_compositor_cs_init_shaders () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#11 0x00007fff811882af in vl_compositor_init () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#12 0x00007fff810692a8 in __vaDriverInit_1_18 () from /gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#13 0x00007fffacfacdb8 in ?? () from /gnu/store/xf6x5qmp75vdnvph041zjdz5779l5q7p-libva-2.18.0/lib/libva.so.2
#14 0x00007fffacfadf26 in vaInitialize () from /gnu/store/xf6x5qmp75vdnvph041zjdz5779l5q7p-libva-2.18.0/lib/libva.so.2
#15 0x00007fff942a7657 in vlc_vaapi_InitializeInstance (o=0x7fff7c001ee0, dpy=0x7fff7c21d6c0, native=native@entry=0x7fff7c2137d0, 
    native_destroy_cb=native_destroy_cb@entry=0x7fff942a64a0 <x11_native_destroy_cb>) at hw/vaapi/vlc_vaapi.c:95
#16 0x00007fff942a7198 in x11_init_vaapi_instance (priv=0x7fff7c2135c0, tc=0x7fff7c212e00) at video_output/opengl/converter_vaapi.c:438
#17 Open (obj=0x7fff7c212e00) at video_output/opengl/converter_vaapi.c:521
#18 0x00007ffff7c9b3f8 in module_load (obj=obj@entry=0x7fff7c212e00, m=m@entry=0x4bb570, init=init@entry=0x7ffff7c9b320 <generic_start>, args=args@entry=0x7fff9478a688)
    at modules/modules.c:183
#19 0x00007ffff7c9b9a4 in vlc_module_load (obj=obj@entry=0x7fff7c212e00, capability=capability@entry=0x7fff94685f34 "glconv", name=0x7ffff7d38487 "", name@entry=0x7fff94685f33 "$glconv", 
    strict=strict@entry=true, probe=probe@entry=0x7ffff7c9b320 <generic_start>) at modules/modules.c:280
#20 0x00007ffff7c9bfa4 in module_need (obj=obj@entry=0x7fff7c212e00, cap=cap@entry=0x7fff94685f34 "glconv", name=name@entry=0x7fff94685f33 "$glconv", strict=strict@entry=true)
    at modules/modules.c:372
#21 0x00007fff9467b46b in opengl_init_program (vgl=vgl@entry=0x7fff7c210690, prgm=prgm@entry=0x7fff7c210998, 
    glexts=glexts@entry=0x7fff7c210d80 "GL_ARB_multisample GL_EXT_abgr GL_EXT_bgra GL_EXT_blend_color GL_EXT_blend_minmax GL_EXT_blend_subtract GL_EXT_copy_texture GL_EXT_subtexture GL_EXT_texture_object GL_EXT_vertex_array GL_EXT_compiled_"..., fmt=fmt@entry=0x7fff7c0014a8, subpics=subpics@entry=false, b_dump_shaders=b_dump_shaders@entry=false)
    at video_output/opengl/vout_helper.c:671
#22 0x00007fff9467c558 in vout_display_opengl_New (fmt=fmt@entry=0x7fff7c0014a8, subpicture_chromas=subpicture_chromas@entry=0x7fff9478a8c0, gl=0x7fff7c001ee0, viewpoint=0x7fff7c00115c)
    at video_output/opengl/vout_helper.c:902
#23 0x00007fff94684166 in Open (obj=0x7fff7c0013c0) at video_output/opengl/display.c:145
#24 0x00007ffff7c9b3f8 in module_load (obj=obj@entry=0x7fff7c0013c0, m=m@entry=0x4bb070, init=init@entry=0x7ffff7c9b320 <generic_start>, args=args@entry=0x7fff9478a9c8)
    at modules/modules.c:183
#25 0x00007ffff7c9b9a4 in vlc_module_load (obj=obj@entry=0x7fff7c0013c0, capability=capability@entry=0x7ffff7d379ca "vout display", name=0x7fff7c001073 "", 
    name@entry=0x7ffff7d48cd9 "$vout", strict=<optimized out>, probe=probe@entry=0x7ffff7c9b320 <generic_start>) at modules/modules.c:280
#26 0x00007ffff7c9bfa4 in module_need (obj=obj@entry=0x7fff7c0013c0, cap=cap@entry=0x7ffff7d379ca "vout display", name=name@entry=0x7ffff7d48cd9 "$vout", strict=<optimized out>)
    at modules/modules.c:372
#27 0x00007ffff7ceb465 in vout_display_New (owner=<synthetic pointer>, cfg=0x7fff7c001130, fmt=0x7fff88065b20, load_module=255, module=0x7ffff7d48cd9 "$vout", obj=0x7fff88065ae0)
    at video_output/display.c:109
#28 DisplayNew (vout=vout@entry=0x7fff88065ae0, source=0x7fff88065b20, state=state@entry=0x7fff9478ac20, module=module@entry=0x7ffff7d48cd9 "$vout", is_splitter=is_splitter@entry=false, 
    double_click_timeout=double_click_timeout@entry=300000, hide_timeout=1000000, owner_ptr=0x0) at video_output/display.c:1198
#29 0x00007ffff7cecf33 in vout_NewDisplay (vout=vout@entry=0x7fff88065ae0, source=<optimized out>, state=state@entry=0x7fff9478ac20, module=module@entry=0x7ffff7d48cd9 "$vout", 
    double_click_timeout=double_click_timeout@entry=300000, hide_timeout=<optimized out>) at video_output/display.c:1255
#30 0x00007ffff7cfd7d8 in vout_OpenWrapper (vout=vout@entry=0x7fff88065ae0, splitter_name=0x0, state=state@entry=0x7fff9478ac20) at video_output/vout_wrapper.c:67
#31 0x00007ffff7ceffd1 in ThreadStart (vout=vout@entry=0x7fff88065ae0, state=0x7fff9478ac20, state@entry=0x0) at video_output/video_output.c:1531
#32 0x00007ffff7cf1ccc in ThreadControl (cmd=..., vout=<optimized out>) at video_output/video_output.c:1686
#33 Thread (object=0x7fff88065ae0) at video_output/video_output.c:1807
#34 0x00007ffff7e053aa in start_thread () from /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6
#35 0x00007ffff7e85f7c in clone3 () from /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6
--8<---------------cut here---------------end--------------->8---

Then I looked at the code of mesa to see what the
'si_create_compute_state', 'tgsi_to_nir' etc were trying to do, and they
were trying to load things from the shader cache. Which is why I tried
deleting the cache, and it worked.


> do you think this bug (is it a bug, right?) needs to be reported
> upstream?

I guess it would be better if the code reading the shader cache was more
robust when reading possibly incompatible or corrupted data. However
I have not tried more recent versions of mesa, maybe they are better at
it...

And it seems that Maxim has already reported the issue upstream,
see <https://issues.guix.gnu.org/63197> and
<https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937>.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]