From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id aNNPFdb/imQ0vwAASxT56A (envelope-from ) for ; Thu, 15 Jun 2023 14:11:02 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id aMHVFNb/imTYdAEAauVa8A (envelope-from ) for ; Thu, 15 Jun 2023 14:11:02 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id F003E1996F for ; Thu, 15 Jun 2023 14:11:01 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q9lo4-0004dk-HD; Thu, 15 Jun 2023 08:10:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q9lnu-0004ad-Nq for guix-devel@gnu.org; Thu, 15 Jun 2023 08:10:18 -0400 Received: from mout01.posteo.de ([185.67.36.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q9lns-00066n-0w for guix-devel@gnu.org; Thu, 15 Jun 2023 08:10:18 -0400 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 7E48324002B for ; Thu, 15 Jun 2023 14:10:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1686831012; bh=hrI9xg4aCGaVeHWRl73hVNVDoc2sf7EPemd6fn/wycI=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From; b=P23S7280+SAN1BBZJpHkccjzpAQCENwJNsJcOwXgHN1oSfHwpdT8tDHLeREiRkKIF QwAUI6zBKiWc6pPOG7bNH/aj+QaGybAw+WbN51dQ28Vaod1ncVNPWQVhs7BWdRsoxn IvfVB+JxTVYtDKKOPYoUA1RGvaKJ9dFGamYkjAJPh7NYWIwD0ugkLswoU2egkvMRLJ xljfvUKXA/IsY5pDq7MSeE0qx9SH+Xo/hmUafB2oFhLPNUauNgEmFkU6UwVkBIFiYy z/Y8neuLRQNWASmWF3NibXxQXm2M4cx/Ayx30aSEH3BtjvM+lMy/2Pj1en5S1s4Y7/ JEz2EM+1ir7Xg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Qhh1v13ZMz9rxS; Thu, 15 Jun 2023 14:10:11 +0200 (CEST) References: <87y1kozvny.fsf@robbyzambito.me> <874jnbn6mk.fsf@kitej> <871qify1i8.fsf@kitej> <87y1kl2f8h.fsf@xelera.eu> From: Guillaume Le Vaillant To: Giovanni Biscuolo Cc: Help Guix , guix-devel@gnu.org Subject: Re: stateful caches (was Re: OBS Studio memory leak) Date: Thu, 15 Jun 2023 11:41:26 +0000 In-reply-to: <87y1kl2f8h.fsf@xelera.eu> Message-ID: <87wn05hqou.fsf@kitej> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=185.67.36.65; envelope-from=glv@posteo.net; helo=mout01.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1686831062; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=4XqPbTCaCQqk+C7Jr8LYXEoU3kmILPYZAjUmWFDLF+k=; b=Z6loeLn7vBiNdkkCoJ0IRmSaRnuUppVr75dZEz5+OdJGtgP4rMf5JgVvAqc++UwNiiQAio 5oS/PrbUT1L6Z9agMat39dPaIu0UIzKeYVY1LT6oDM6Kn90EnXW/rdbgxPjjU+gfThdnsj 2qHXNhkQvWLiriW9WMRX/gduvkgvaFMm8zzjw1/6v6t+/jiygTJhUtxLrfI45LAFc1L2N3 d7B0F+wjcMCrLaM2jRfJllUD4Et8xOqFonq9ZX7xJ7XEM7gKZtEf6KENw3jmCRbnpqfMV9 LRoNcn8jB8nDhVXWryayIJZ+RTJkTdOHB9Xpj2A09QQF0Wuez8ci12vGVUFUGA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=P23S7280; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1686831062; a=rsa-sha256; cv=none; b=E63CjspvzlcIXy1PBD2k6P1SBBn4AjKN2ER033bNDp1JXv75qEv1jmbilFRdxHqlUO+3h4 Hh9r1vJDwQFqqOnVnBwsWYfY88vxWmrGMHKXFHJrUK/w3PijunwrRIwMf9YU6uODo96MA1 tLs7QRMbzIfZPjnQpe+pjDw53DNWDm84Fj3OylxXvU8kfMrv6oo9RMaAcVuAgaidtKr9Ev u+rliovKPaRnncT8LZoEK9PGxVmuku/UA4bLHFrgfH8NSMCdRxh/IXEUQRQJRuv/cPRqJE Un/kpcfivAryWHR9tjmyTLtWKJgZsszOFgidKD0eCbiMbBsOgzApaDYwuTxjcA== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -7.79 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=P23S7280; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: F003E1996F X-Spam-Score: -7.79 X-TUID: wyygBO3FdzRs --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Giovanni Biscuolo skribis: > Hi Guillaume Le Vaillant and Guix Devels, > > sorry for cross-posting but IMHO the workaround you found [1] for the mem= ory > leak affecting a number of media processing applications is of interest > for many people potentially not subscribed to help-guix > > AFAIK this was not filed as a Guix bug > > Guillaume Le Vaillant writes: > >> Ott Joon skribis: >> >>> Hey >>> >>> Tried the same thing in VLC and it freezes on GPU accel and starts >>> leaking memory while also becoming hard to kill. Maybe this also >>> explains why some mpv GPU accel settings don't work also in the exact >>> same way. I have an AMD RX 6900 XT on this machine. > > [...] > >> It looks like an issue with the shader cache of mesa. >> After clearing it, I don't see the memory leak anymore. > > good catch: please can you tell us how you managed to spot that problem? > Did you straced it or did yoy find a related mesa bug report? I used gdb on versions of mesa and vlc with debug symbols: =2D-8<---------------cut here---------------start------------->8--- guix build --with-debug-info=3Dmesa --with-debug-info=3Dvlc vlc gdb /gnu/store/...-vlc-3.0.18/bin/.vlc-real (gdb) run some-video.mkv =2D-8<---------------cut here---------------end--------------->8--- Then I sent a SIGSTOP signal to the vlc process, and in gdb I looked at the backtrace of all the threads of vlc. There was only one thread allocating things: =2D-8<---------------cut here---------------start------------->8--- Thread 34 #0 0x00007ffff7e127c7 in sysmalloc () from /gnu/store/gsjczqir1wbz8p770znd= rpw4rnppmxi3-glibc-2.35/lib/libc.so.6 #1 0x00007ffff7e13cfb in _int_malloc () from /gnu/store/gsjczqir1wbz8p770z= ndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6 #2 0x00007ffff7e145b5 in malloc () from /gnu/store/gsjczqir1wbz8p770zndrpw= 4rnppmxi3-glibc-2.35/lib/libc.so.6 #3 0x00007fff810822b1 in ralloc_size () from /gnu/store/5gwcwadzfm48j9cm2b= 7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #4 0x00007fff81710b2c in read_constant () from /gnu/store/5gwcwadzfm48j9cm= 2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #5 0x00007fff817110f2 in read_var_list () from /gnu/store/5gwcwadzfm48j9cm= 2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #6 0x00007fff81711865 in nir_deserialize () from /gnu/store/5gwcwadzfm48j9= cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #7 0x00007fff8162c874 in tgsi_to_nir () from /gnu/store/5gwcwadzfm48j9cm2b= 7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #8 0x00007fff8146cba1 in si_create_compute_state () from /gnu/store/5gwcwa= dzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #9 0x00007fff8118e1ba in vl_compositor_cs_create_shader () from /gnu/store= /5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #10 0x00007fff8118e800 in vl_compositor_cs_init_shaders () from /gnu/store/= 5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #11 0x00007fff811882af in vl_compositor_init () from /gnu/store/5gwcwadzfm4= 8j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #12 0x00007fff810692a8 in __vaDriverInit_1_18 () from /gnu/store/5gwcwadzfm= 48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so #13 0x00007fffacfacdb8 in ?? () from /gnu/store/xf6x5qmp75vdnvph041zjdz5779= l5q7p-libva-2.18.0/lib/libva.so.2 #14 0x00007fffacfadf26 in vaInitialize () from /gnu/store/xf6x5qmp75vdnvph0= 41zjdz5779l5q7p-libva-2.18.0/lib/libva.so.2 #15 0x00007fff942a7657 in vlc_vaapi_InitializeInstance (o=3D0x7fff7c001ee0,= dpy=3D0x7fff7c21d6c0, native=3Dnative@entry=3D0x7fff7c2137d0,=20 native_destroy_cb=3Dnative_destroy_cb@entry=3D0x7fff942a64a0 ) at hw/vaapi/vlc_vaapi.c:95 #16 0x00007fff942a7198 in x11_init_vaapi_instance (priv=3D0x7fff7c2135c0, t= c=3D0x7fff7c212e00) at video_output/opengl/converter_vaapi.c:438 #17 Open (obj=3D0x7fff7c212e00) at video_output/opengl/converter_vaapi.c:521 #18 0x00007ffff7c9b3f8 in module_load (obj=3Dobj@entry=3D0x7fff7c212e00, m= =3Dm@entry=3D0x4bb570, init=3Dinit@entry=3D0x7ffff7c9b320 , = args=3Dargs@entry=3D0x7fff9478a688) at modules/modules.c:183 #19 0x00007ffff7c9b9a4 in vlc_module_load (obj=3Dobj@entry=3D0x7fff7c212e00= , capability=3Dcapability@entry=3D0x7fff94685f34 "glconv", name=3D0x7ffff7d= 38487 "", name@entry=3D0x7fff94685f33 "$glconv",=20 strict=3Dstrict@entry=3Dtrue, probe=3Dprobe@entry=3D0x7ffff7c9b320 ) at modules/modules.c:280 #20 0x00007ffff7c9bfa4 in module_need (obj=3Dobj@entry=3D0x7fff7c212e00, ca= p=3Dcap@entry=3D0x7fff94685f34 "glconv", name=3Dname@entry=3D0x7fff94685f33= "$glconv", strict=3Dstrict@entry=3Dtrue) at modules/modules.c:372 #21 0x00007fff9467b46b in opengl_init_program (vgl=3Dvgl@entry=3D0x7fff7c21= 0690, prgm=3Dprgm@entry=3D0x7fff7c210998,=20 glexts=3Dglexts@entry=3D0x7fff7c210d80 "GL_ARB_multisample GL_EXT_abgr = GL_EXT_bgra GL_EXT_blend_color GL_EXT_blend_minmax GL_EXT_blend_subtract GL= _EXT_copy_texture GL_EXT_subtexture GL_EXT_texture_object GL_EXT_vertex_arr= ay GL_EXT_compiled_"..., fmt=3Dfmt@entry=3D0x7fff7c0014a8, subpics=3Dsubpic= s@entry=3Dfalse, b_dump_shaders=3Db_dump_shaders@entry=3Dfalse) at video_output/opengl/vout_helper.c:671 #22 0x00007fff9467c558 in vout_display_opengl_New (fmt=3Dfmt@entry=3D0x7fff= 7c0014a8, subpicture_chromas=3Dsubpicture_chromas@entry=3D0x7fff9478a8c0, g= l=3D0x7fff7c001ee0, viewpoint=3D0x7fff7c00115c) at video_output/opengl/vout_helper.c:902 #23 0x00007fff94684166 in Open (obj=3D0x7fff7c0013c0) at video_output/openg= l/display.c:145 #24 0x00007ffff7c9b3f8 in module_load (obj=3Dobj@entry=3D0x7fff7c0013c0, m= =3Dm@entry=3D0x4bb070, init=3Dinit@entry=3D0x7ffff7c9b320 , = args=3Dargs@entry=3D0x7fff9478a9c8) at modules/modules.c:183 #25 0x00007ffff7c9b9a4 in vlc_module_load (obj=3Dobj@entry=3D0x7fff7c0013c0= , capability=3Dcapability@entry=3D0x7ffff7d379ca "vout display", name=3D0x7= fff7c001073 "",=20 name@entry=3D0x7ffff7d48cd9 "$vout", strict=3D, probe=3D= probe@entry=3D0x7ffff7c9b320 ) at modules/modules.c:280 #26 0x00007ffff7c9bfa4 in module_need (obj=3Dobj@entry=3D0x7fff7c0013c0, ca= p=3Dcap@entry=3D0x7ffff7d379ca "vout display", name=3Dname@entry=3D0x7ffff7= d48cd9 "$vout", strict=3D) at modules/modules.c:372 #27 0x00007ffff7ceb465 in vout_display_New (owner=3D, cf= g=3D0x7fff7c001130, fmt=3D0x7fff88065b20, load_module=3D255, module=3D0x7ff= ff7d48cd9 "$vout", obj=3D0x7fff88065ae0) at video_output/display.c:109 #28 DisplayNew (vout=3Dvout@entry=3D0x7fff88065ae0, source=3D0x7fff88065b20= , state=3Dstate@entry=3D0x7fff9478ac20, module=3Dmodule@entry=3D0x7ffff7d48= cd9 "$vout", is_splitter=3Dis_splitter@entry=3Dfalse,=20 double_click_timeout=3Ddouble_click_timeout@entry=3D300000, hide_timeou= t=3D1000000, owner_ptr=3D0x0) at video_output/display.c:1198 #29 0x00007ffff7cecf33 in vout_NewDisplay (vout=3Dvout@entry=3D0x7fff88065a= e0, source=3D, state=3Dstate@entry=3D0x7fff9478ac20, module= =3Dmodule@entry=3D0x7ffff7d48cd9 "$vout",=20 double_click_timeout=3Ddouble_click_timeout@entry=3D300000, hide_timeou= t=3D) at video_output/display.c:1255 #30 0x00007ffff7cfd7d8 in vout_OpenWrapper (vout=3Dvout@entry=3D0x7fff88065= ae0, splitter_name=3D0x0, state=3Dstate@entry=3D0x7fff9478ac20) at video_ou= tput/vout_wrapper.c:67 #31 0x00007ffff7ceffd1 in ThreadStart (vout=3Dvout@entry=3D0x7fff88065ae0, = state=3D0x7fff9478ac20, state@entry=3D0x0) at video_output/video_output.c:1= 531 #32 0x00007ffff7cf1ccc in ThreadControl (cmd=3D..., vout=3D)= at video_output/video_output.c:1686 #33 Thread (object=3D0x7fff88065ae0) at video_output/video_output.c:1807 #34 0x00007ffff7e053aa in start_thread () from /gnu/store/gsjczqir1wbz8p770= zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6 #35 0x00007ffff7e85f7c in clone3 () from /gnu/store/gsjczqir1wbz8p770zndrpw= 4rnppmxi3-glibc-2.35/lib/libc.so.6 =2D-8<---------------cut here---------------end--------------->8--- Then I looked at the code of mesa to see what the 'si_create_compute_state', 'tgsi_to_nir' etc were trying to do, and they were trying to load things from the shader cache. Which is why I tried deleting the cache, and it worked. > do you think this bug (is it a bug, right?) needs to be reported > upstream? I guess it would be better if the code reading the shader cache was more robust when reading possibly incompatible or corrupted data. However I have not tried more recent versions of mesa, maybe they are better at it... And it seems that Maxim has already reported the issue upstream, see and . --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCZIr/oQ8cZ2x2QHBvc3Rl by5uZXQACgkQa+ggit8h/j/duAD9HgHomp0dkdEQgE+OgS0iDnTbjsyLqXT6CDzp Fe9JtjgA/0CS4wH/8Qvt8/YSMwI0Q89oeRO/gPxe8lMVSCzR1n2E =wk6M -----END PGP SIGNATURE----- --=-=-=--