From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Konstantin Kharlamov Newsgroups: gmane.emacs.bugs Subject: bug#45200: Memory leaks: (garbage-collect) fails to reclaim memory Date: Sun, 24 Jan 2021 23:00:50 +0300 Message-ID: <5c5696670dea31a4647c901be1f87ddb1474f820.camel@yandex.ru> References: <2a4f3f31f30db28387055821a8edf44fd1d66ea5.camel@yandex.ru> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="16997"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Evolution 3.38.3 Cc: 45200@debbugs.gnu.org To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun Jan 24 21:02:20 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l3laV-0004Ij-PW for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 24 Jan 2021 21:02:19 +0100 Original-Received: from localhost ([::1]:50204 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l3laU-0005Lg-Rx for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 24 Jan 2021 15:02:18 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:36796) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l3laF-0005Jy-Pa for bug-gnu-emacs@gnu.org; Sun, 24 Jan 2021 15:02:03 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:54056) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l3laF-0004YY-5C for bug-gnu-emacs@gnu.org; Sun, 24 Jan 2021 15:02:03 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l3laE-0003VB-2u for bug-gnu-emacs@gnu.org; Sun, 24 Jan 2021 15:02:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Konstantin Kharlamov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 24 Jan 2021 20:02:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45200 X-GNU-PR-Package: emacs Original-Received: via spool by 45200-submit@debbugs.gnu.org id=B45200.161151846113378 (code B ref 45200); Sun, 24 Jan 2021 20:02:02 +0000 Original-Received: (at 45200) by debbugs.gnu.org; 24 Jan 2021 20:01:01 +0000 Original-Received: from localhost ([127.0.0.1]:37369 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l3lZF-0003Te-Ie for submit@debbugs.gnu.org; Sun, 24 Jan 2021 15:01:01 -0500 Original-Received: from forward101j.mail.yandex.net ([5.45.198.241]:49730) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l3lZD-0003TG-Gb for 45200@debbugs.gnu.org; Sun, 24 Jan 2021 15:01:01 -0500 Original-Received: from myt2-4597fff41cb2.qloud-c.yandex.net (myt2-4597fff41cb2.qloud-c.yandex.net [IPv6:2a02:6b8:c00:3b2c:0:640:4597:fff4]) by forward101j.mail.yandex.net (Yandex) with ESMTP id 587741BE022B; Sun, 24 Jan 2021 23:00:52 +0300 (MSK) Original-Received: from myt6-efff10c3476a.qloud-c.yandex.net (myt6-efff10c3476a.qloud-c.yandex.net [2a02:6b8:c12:13a3:0:640:efff:10c3]) by myt2-4597fff41cb2.qloud-c.yandex.net (mxback/Yandex) with ESMTP id YPeZnxVA9D-0qH8wQt8; Sun, 24 Jan 2021 23:00:52 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1611518452; bh=DGHB7wTmROhuQJyXdW2kGLCrWOr9FKnXoWpERvnKdyo=; h=In-Reply-To:Cc:To:From:Subject:Message-ID:References:Date; b=p8evXnbDVwjP27fx+KfRrHA0PUuZqm2caDNYaOyqdXfJGqJ5kqKxm5CaJYuVlCQbh 10p2shh2hBVMMcm6Xiz1A3vwMdvSG7rBog0XDJPurLdvsOm6wxdPTrHJ5XF9mJae60 1H/HtZ4FClOfNWnRRaOr0w0nv0M+UieaX0b1p5e0= Authentication-Results: myt2-4597fff41cb2.qloud-c.yandex.net; dkim=pass header.i=@yandex.ru Original-Received: by myt6-efff10c3476a.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id wghFCbz68h-0pHiXOJg; Sun, 24 Jan 2021 23:00:51 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:198523 Archived-At: On Sun, 2021-01-24 at 14:12 -0500, Stefan Monnier wrote: > > I'm fine with Emacs possibly keeping a dozen of megabytes for personal > > use. The problem though is that the issue easily manifests itself in > > hundreds of megabytes, as can be seen with the testcase marked as "Message > > #11" in web-ui. > > I don't think the exact number matters: you were willing to make use of > *any* amount of extra memory by delaying all GC until idle time. > The fact that the GC you finally do after that long wait doesn't recover > that memory is not a failure in my book: there's no reason Emacs or > glibc should presume that you won't be reusing just as much heap space > before the next GC. > > IOW in my book, you got what you asked for ;-) This doesn't look right. I mean, sure, I might need the heap again. Or I might not. I don't know. Neither you can. Nobody knows. If glibc can predict the future then fine. But my experience as part of this report suggests glibc can't. Glibc could for example use statistics of previous allocation patterns to see if it's needed to release memory and how much, but this is not what happens. And I am not the only dissatisfied user. Ruby for example too¹. Also, many people experience strange memory usage grow with Qutebrowser that nobody knows where it's coming from; and after today's research I suspect Glibc is the culprit (I don't have yet enough evidence because my Qutebrowser instance doesn't have much uptime ATM, but attaching to it with gdb and calling `malloc_trim` inside it already freed ≈40M of memory to me today). Glib also seems affected². With so many unhappy users, do you still think nothing wrong with the glibc allocator? It seems to me, we're doing perfectly valid usecase: we used memory, we don't need it anymore, we free it. If somebody ever notes that malloc/free cycle takes too much time for them, then they know they need to optimize the specific codepath where this happens. It's a nice bonus if Glibc could do malloc/free faster, but not so much if this causes other problems, especially given it's a core system library. > BTW, I wish `malloc_trim` could be passed an argument that says > something like "release about half of the free memory", which would > provide a better balance between "hoard free memory indefinitely" and > "constantly free&reallocate memory from the OS". > > Also, the manpage of mallopt suggests that glibc should automatically > do the trim on its own every once in a while anyway, so maybe the > problem won't be solved by `malloc_trim`. Yeah, I've seen that too. Idk why it doesn't work, but that `malloc_trim()` does I know because I tested it with the testcase where previously Emacs was never returning ≈200M of memory. 1: https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#investigating-fragmentation-at-the-memory-allocator-level 2: https://gitlab.gnome.org/GNOME/glib/-/issues/2057