From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Newsgroups: gmane.emacs.bugs Subject: bug#45200: [PATCH] Force Glibc to free the memory freed Date: Wed, 19 May 2021 00:11:32 -0400 Message-ID: References: <83k0rz21dw.fsf@gnu.org> <331805c74fc5d3d412dd2065030b11fa3343710d.camel@yandex.ru> <8a91fc16f93298bca1281b43d6821ae3621376dc.camel@yandex.ru> <7ffacc5f-fc0e-a5f8-104d-2c0ae8e06953@gmx.at> <3f4f8b3034e9f52f48ec520f2732b1687bb24dbc.camel@yandex.ru> <31608795-6155-c7c9-7d94-6024adb0a3b9@gmx.at> <09cc70f14e90f6b13b51b0536fae76e87dfe3f42.camel@yandex.ru> <55be0318-c907-bf9b-d644-d88abb816871@gmx.at> <35163027-a5b7-4a6c-6700-69d34fecf451@gmx.at> <824625557ce288b0cbc3edd66ff730afd479b511.camel@yandex.ru> Reply-To: Stefan Monnier Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="12023"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: carlos@redhat.com, martin rudalics , dj@redhat.com, fweimer@redhat.com, 45200@debbugs.gnu.org To: Konstantin Kharlamov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed May 19 06:12:11 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ljDZ4-0002wb-Sh for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 19 May 2021 06:12:11 +0200 Original-Received: from localhost ([::1]:57792 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ljDZ3-0008ST-EO for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 19 May 2021 00:12:09 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:35254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ljDYw-0008SK-GS for bug-gnu-emacs@gnu.org; Wed, 19 May 2021 00:12:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:46182) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ljDYw-0006FF-6v for bug-gnu-emacs@gnu.org; Wed, 19 May 2021 00:12:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ljDYw-0007FZ-11 for bug-gnu-emacs@gnu.org; Wed, 19 May 2021 00:12:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 19 May 2021 04:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45200 X-GNU-PR-Package: emacs Original-Received: via spool by 45200-submit@debbugs.gnu.org id=B45200.162139750827849 (code B ref 45200); Wed, 19 May 2021 04:12:01 +0000 Original-Received: (at 45200) by debbugs.gnu.org; 19 May 2021 04:11:48 +0000 Original-Received: from localhost ([127.0.0.1]:57728 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljDYh-0007F7-RV for submit@debbugs.gnu.org; Wed, 19 May 2021 00:11:48 -0400 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:12125) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljDYf-0007Eu-Pe for 45200@debbugs.gnu.org; Wed, 19 May 2021 00:11:47 -0400 Original-Received: from pmg2.iro.umontreal.ca (localhost.localdomain [127.0.0.1]) by pmg2.iro.umontreal.ca (Proxmox) with ESMTP id 84455807A1; Wed, 19 May 2021 00:11:40 -0400 (EDT) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg2.iro.umontreal.ca (Proxmox) with ESMTP id AD18680792; Wed, 19 May 2021 00:11:38 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1621397498; bh=3DKKlUFcRxJ6kEgQVaNvE1l2P7Vc3sJ2HfiSiNxQQOw=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=AzYi9RMBEtePMhCq3sI7ST2TkPDAnbJh8MWZsI7AL/vAE75fPe7YAY2iFELpwMsIm cH6U/2XR4LCWpb076S4/1X6L8eMe6qlCkHWg/Dlkk6JWVp8/ewrd9daYHc4rpHrRO1 q8UbbIWBiaHvcVyx1pPXS9Ewc55WTAqGV8QxZo4SfhN8QzpbB8TJoV+ut/B22LUCkg jyrlXZ2+pCrCqZ46bnUmPjVYvjFbN8B0Mpv7jI2UDEtV0RnYgYgDaq/8Ic3hGmGE7n pqRzHtDzB1fUcrmyZhSsyJVtsyGmL+w5Nep1eGBBMg3Boa+lx5ZbhyWw24wcCNSIB8 9DKi77VLCWVSQ== Original-Received: from alfajor (76-10-140-76.dsl.teksavvy.com [76.10.140.76]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 5F31F120429; Wed, 19 May 2021 00:11:38 -0400 (EDT) In-Reply-To: (Konstantin Kharlamov's message of "Tue, 18 May 2021 23:12:53 +0300") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:206854 Archived-At: > Okay, apparently I'm not the only one affected by the memory problem=C2= =B9, so I'll > try to make the necessary change for it to go over the finish line. Sadly, malloc_trim does not satisfy the request "a GC that actually releases the memory back to the system": we do release it back to glibc but glibc only does it when we're lucky enough that the memory we released is all at the very end of the heap (and that constraint apparently also applies to `mallov_trim`). A concurrent GC would be great, yes. XEmacs had that, pretty much (not concurrent, only incremental, but it should not be terribly hard to push it over the edge and make it concurrent). > I have a question though. So, wrapping `malloc_trim` in elisp and adding = it to > idle with `(run-with-idle-timer)` sounds simple. But, where in code shoul= d I do > the `(run-with-idle-timer)` call, i.e. so it's called as is emacs is gett= ing > initialized? I'd oppose calling `malloc-trim` in the default config. So the answer would be "in your ~/.emacs" (presumably next to the code that changes the GC config vars ;-) Stefan > > 1: > https://www.reddit.com/r/emacs/comments/n13v5l/what_is_the_next_big_featu= re_after_native_comp/gwc5g77/?context=3D3 > > On Wed, 2021-02-03 at 11:02 -0500, Stefan Monnier wrote: >> > To answer you question in another email about memory benefits given de= fault >> > Emacs settings: well, today I tried creating a testcase that would rep= roduce >> > problem with default settings, but haven't really succeeded at it.=C2= =A0 I have >> > a testcase where Emacs without the patch has =E2=89=8820M more memory = than the one >> > with the patch, but that's pretty small difference, and offhand I didn= 't >> > manage to get it increased any further. >>=20 >> Thanks.=C2=A0 At least that seems to indicate that glibc does its job >> properly for the way we normally use it. >>=20 >> > So, I'm thinking of wiring the functional of memory trimming to on-idle >> > hook, without possibility to disable it. >>=20 >> That seems hard to do (luckily), since AFAICT idle hooks only exist via >> `run-with-idle-timer` and those can always be disabled with things like >> `cancel-timer`. >>=20 >> > Given how small performance impact in this case would be, I see no >> > reason to even implement an option to disable it. >> > Thoughts? >>=20 >> My main thought is that if `malloc_trim` indeed makes a significant >> difference, it's a sign that Emacs's own code did its job (it called >> `free` as it should) and that the problem is in how glibc decided not >> to return the memory to the OS. >>=20 >> That's a behavior that can (and will) change over time outside of >> our control.=C2=A0 So calling `malloc_trim` every time I stop typing for= 10s, >> just on account of "maybe glibc didn't reclaim quite as much memory as >> we'd like this time" doesn't sound very appealing to me. >>=20 >> It sounds like an ad-hoc quick hack.=C2=A0 I love being able to use ad-h= oc >> quick hacks, but I don't like enabling such things by default when the >> only use-cases where they're known to be useful are fairly specialized >> (and discouraged by Emacs maintainers ;-) >>=20 >> So I think we need more info: do the glibc maintainers consider it >> normal for glibc to behave this way?=C2=A0 Why does it behave this way? >> Would the equivalent of `malloc_trim` happen anyway "at some point in >> the future"?=C2=A0 E.g. If you create a test case where you disable GC, = let >> the memory use grow to 1GB, then reset the GC vars to their default and >> keep using Emacs modestly, will the memory ever be returned to the OS or >> is an explicit call to `malloc_trim` really indispensable? >>=20 >> But until we get all the answers to these questions, we can already >> install the code that exposes `malloc_trim` to ELisp. >>=20 >>=20 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Stefan >>=20