From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Konstantin Kharlamov Newsgroups: gmane.emacs.bugs Subject: bug#45200: [PATCH] Force Glibc to free the memory freed Date: Wed, 03 Feb 2021 19:35:08 +0300 Message-ID: <6b1ddaa05952c3d942ca74f187427c2319605621.camel@yandex.ru> References: <83k0rz21dw.fsf@gnu.org> <331805c74fc5d3d412dd2065030b11fa3343710d.camel@yandex.ru> <8a91fc16f93298bca1281b43d6821ae3621376dc.camel@yandex.ru> <7ffacc5f-fc0e-a5f8-104d-2c0ae8e06953@gmx.at> <3f4f8b3034e9f52f48ec520f2732b1687bb24dbc.camel@yandex.ru> <31608795-6155-c7c9-7d94-6024adb0a3b9@gmx.at> <09cc70f14e90f6b13b51b0536fae76e87dfe3f42.camel@yandex.ru> <55be0318-c907-bf9b-d644-d88abb816871@gmx.at> <35163027-a5b7-4a6c-6700-69d34fecf451@gmx.at> <824625557ce288b0cbc3edd66ff730afd479b511.camel@yandex.ru> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="34824"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Evolution 3.38.3 Cc: carlos@redhat.com, fweimer@redhat.com, dj@redhat.com, 45200@debbugs.gnu.org To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Feb 03 17:49:51 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l7LLj-0008yf-0L for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 03 Feb 2021 17:49:51 +0100 Original-Received: from localhost ([::1]:52670 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7LLi-0006TO-3K for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 03 Feb 2021 11:49:50 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:55584) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7L8M-0003Px-Um for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 11:36:05 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:55387) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l7L8L-0004VF-N0 for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 11:36:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l7L8L-0005vZ-Jk for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 11:36:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Konstantin Kharlamov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 03 Feb 2021 16:36:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45200 X-GNU-PR-Package: emacs Original-Received: via spool by 45200-submit@debbugs.gnu.org id=B45200.161237012022723 (code B ref 45200); Wed, 03 Feb 2021 16:36:01 +0000 Original-Received: (at 45200) by debbugs.gnu.org; 3 Feb 2021 16:35:20 +0000 Original-Received: from localhost ([127.0.0.1]:38700 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l7L7g-0005uR-DR for submit@debbugs.gnu.org; Wed, 03 Feb 2021 11:35:20 -0500 Original-Received: from forward105o.mail.yandex.net ([37.140.190.183]:60404) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l7L7d-0005u3-7D for 45200@debbugs.gnu.org; Wed, 03 Feb 2021 11:35:19 -0500 Original-Received: from sas1-f45a06513e06.qloud-c.yandex.net (sas1-f45a06513e06.qloud-c.yandex.net [IPv6:2a02:6b8:c08:b315:0:640:f45a:651]) by forward105o.mail.yandex.net (Yandex) with ESMTP id A5CF942004CB; Wed, 3 Feb 2021 19:35:09 +0300 (MSK) Original-Received: from sas2-e7f6fb703652.qloud-c.yandex.net (sas2-e7f6fb703652.qloud-c.yandex.net [2a02:6b8:c14:4fa6:0:640:e7f6:fb70]) by sas1-f45a06513e06.qloud-c.yandex.net (mxback/Yandex) with ESMTP id 00bDFJtm6O-Z9I0IXc2; Wed, 03 Feb 2021 19:35:09 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1612370109; bh=g1p/+di+AOlqV0fs18sOxT02JmGa6LIz8fsJkvSIldo=; h=In-Reply-To:Cc:To:From:Subject:Message-ID:References:Date; b=Rtz7SUv2eWRHEuFQkGG27TJfAF/PaRrfp1pESMhSnntq2Pt86mivAakWgqE1wDSfU /MOu+D9/zXGSb3j+GmtbywLKv6dcZBoky8otHp/1/HNUPo8iiQ5r0KDjEQJ1QBZCWY xLZh6Z6yCBfWiirp04iFxHKbUSgb0kyKxN5g8xR8= Authentication-Results: sas1-f45a06513e06.qloud-c.yandex.net; dkim=pass header.i=@yandex.ru Original-Received: by sas2-e7f6fb703652.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id JAqwHAvyMr-Z8nSY1KT; Wed, 03 Feb 2021 19:35:08 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:199212 Archived-At: On Wed, 2021-02-03 at 11:02 -0500, Stefan Monnier wrote: > > To answer you question in another email about memory benefits given default > > Emacs settings: well, today I tried creating a testcase that would reproduce > > problem with default settings, but haven't really succeeded at it.  I have > > a testcase where Emacs without the patch has ≈20M more memory than the one > > with the patch, but that's pretty small difference, and offhand I didn't > > manage to get it increased any further. > > Thanks.  At least that seems to indicate that glibc does its job > properly for the way we normally use it. > > > So, I'm thinking of wiring the functional of memory trimming to on-idle > > hook, without possibility to disable it. > > That seems hard to do (luckily), since AFAICT idle hooks only exist via > `run-with-idle-timer` and those can always be disabled with things like > `cancel-timer`. Ah, okay then. I just didn't know how on-idle code should be implemented; if it would be just making a ELisp wrapper and adding it to the hook, which people can then remove if they want to, that's fine. I just thought there's some additional work required to make it "disableable", I'm not opposed to the idea of it being so per se. > > Given how small performance impact in this case would be, I see no > > reason to even implement an option to disable it. > > Thoughts? > > My main thought is that if `malloc_trim` indeed makes a significant > difference, it's a sign that Emacs's own code did its job (it called > `free` as it should) and that the problem is in how glibc decided not > to return the memory to the OS. > > That's a behavior that can (and will) change over time outside of > our control.  So calling `malloc_trim` every time I stop typing for 10s, > just on account of "maybe glibc didn't reclaim quite as much memory as > we'd like this time" doesn't sound very appealing to me. > > It sounds like an ad-hoc quick hack.  I love being able to use ad-hoc > quick hacks, but I don't like enabling such things by default when the > only use-cases where they're known to be useful are fairly specialized > (and discouraged by Emacs maintainers ;-) I totally agree with you, and I too consider it a hack. > So I think we need more info: do the glibc maintainers consider it > normal for glibc to behave this way?  Why does it behave this way? Very good question! I hope Glibc mainainers that are on CC list will be able to answer. Even though I created a report on Glibc just some months ago, the problem per se existed for a long time. And I've seen Carlos O'Donell leaving a small comment on a similar issue with Ruby 2 years ago, which implies they're aware of this situation. > Would the equivalent of `malloc_trim` happen anyway "at some point in > the future"?  E.g. If you create a test case where you disable GC, let > the memory use grow to 1GB, then reset the GC vars to their default and > keep using Emacs modestly, will the memory ever be returned to the OS or > is an explicit call to `malloc_trim` really indispensable? No, the memory will never be returned to OS. I can tell that right away, because the only difference would be `free()` getting called more often. I think it is worth mentioning here that Glibc usually does return memory to the OS without any need in malloc_trim(0). What happens in affected applications (such as here) is that an application stumbles upon a very special allocation pattern, which kinda breaks Glibc algorithms of returning memory. That's btw to the point of whether this GLibc behavior is a bug: well, sometimes it works, sometimes it doesn't — doesn't look like purposeful behavior to me :) > But until we get all the answers to these questions, we can already > install the code that exposes `malloc_trim` to ELisp. > > >         Stefan >