From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: DJ Delorie Newsgroups: gmane.emacs.bugs Subject: bug#45200: [PATCH] Force Glibc to free the memory freed Date: Wed, 03 Feb 2021 15:42:11 -0500 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="35607"; mail-complaints-to="usenet@ciao.gmane.io" Cc: fweimer@redhat.com, carlos@redhat.com, hi-angel@yandex.ru, 45200@debbugs.gnu.org To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Feb 03 21:43:15 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l7OzZ-00097A-54 for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 03 Feb 2021 21:43:13 +0100 Original-Received: from localhost ([::1]:60612 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7OzY-0000Xj-6I for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 03 Feb 2021 15:43:12 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:36268) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7OzO-0000XK-OY for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 15:43:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:55667) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l7OzO-0002cJ-Hb for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 15:43:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l7OzO-0003sO-EN for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 15:43:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: DJ Delorie Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 03 Feb 2021 20:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45200 X-GNU-PR-Package: emacs Original-Received: via spool by 45200-submit@debbugs.gnu.org id=B45200.161238494414855 (code B ref 45200); Wed, 03 Feb 2021 20:43:02 +0000 Original-Received: (at 45200) by debbugs.gnu.org; 3 Feb 2021 20:42:24 +0000 Original-Received: from localhost ([127.0.0.1]:38980 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l7Oyl-0003rW-PX for submit@debbugs.gnu.org; Wed, 03 Feb 2021 15:42:24 -0500 Original-Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:59780) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l7Oyi-0003rN-Uv for 45200@debbugs.gnu.org; Wed, 03 Feb 2021 15:42:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612384940; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=SchxRDK8NZSNl+opd9cfPeLz0psV4KLmDWg5JsEEbtk=; b=I63cvQYJN+7tu0RfXYGRMmDA7a8IVP/IXQZ5+1cvLbQ5r9gQU7f+tdWH3EL1tCzOXuANzx JVWu7REnRUx8iww4ALD6kW3CRhJoG1+UQnnWrw6PtoZcHBBqLqGPiyz2SDdoMOlVtdXm43 eCDAC+soyMoBKRONsLlvDyZixPlRxaM= Original-Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-55-baaWDf1oPR6Q_PrEVon4Nw-1; Wed, 03 Feb 2021 15:42:18 -0500 X-MC-Unique: baaWDf1oPR6Q_PrEVon4Nw-1 Original-Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id F171B801967; Wed, 3 Feb 2021 20:42:16 +0000 (UTC) Original-Received: from greed.delorie.com (ovpn-114-77.rdu2.redhat.com [10.10.114.77]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9DDC358899; Wed, 3 Feb 2021 20:42:13 +0000 (UTC) Original-Received: from greed.delorie.com.redhat.com (localhost [127.0.0.1]) by greed.delorie.com (8.14.7/8.14.7) with ESMTP id 113KgB2q001712; Wed, 3 Feb 2021 15:42:11 -0500 In-Reply-To: (message from Stefan Monnier on Wed, 03 Feb 2021 15:24:23 -0500) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dj@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:199245 Archived-At: Stefan Monnier writes: > I understand that as well. But I'm wondering why glibc is willing to > keep *indefinitely* an unused 200MB of memory, which is more than double > the mount of memory in use for the rest of the application's life. To be blunt, 200Mb is peanuts compared to some applications, and it's *nothing* compared to an enterprise application. Keeping 200M around to quickly satisfy memory requests of various sizes (not all cached chunks are the same size) is IMHO reasonable. > I mean I understand that you can't predict the future, but I expected > that "at some point" glibc should decide that those 200MB have been left > unused for long enough that they deserve to be returned to the OS. Where will we store that lifetime information? Yet another word of memory used, yet another syscall to check the time? Or completely reorganize the malloc internals into an LRU system so we can peel off the old end, at the expense of performance? I agree that we could do better at detecting long-unused chunks, but it's expensive (in terms of both development and runtime) to do so, and typically at the expense of some other desired metric. I would ask the Emacs devs why they wait until gc to free() memory instead of keeping track of uses more accurately and free()ing it right away. It's a similar type of compromise. > The doc of `malloc_trim` suggests it's occasionally called by `free` and > `mallopt` suggests via `M_TRIM_THRESHOLD` that there's a limit to how > much extra spare memory glibc keeps around, so this suggests that indeed > memory is trimmed "every once in a while". Only when the available memory is "at the top of the heap". Most cached memory is not; one unfree'd chunk at the top of the heap can keep the whole heap in memory. We used to have code that munmap()'d large "holes" in the cache, but the resulting performance was horrible. > so my guess is that in Emacs's case those 200MB of extra memory are > *not* contiguous? Would that be it? Right, or just not at the top of the heap. > Is there a way for Emacs to ask malloc how much memory (or some > approximation of it) would be recovered by `malloc_trim`? Not easily. You can call mallinfo2() before and after a malloc_trim(), but then you'd have *done* the trim ;-) > In our normal use, glibc's tuning works fine. We're just seeing some > corner case behaviors which seem clearly undesirable, so I think it > would make sense to try and arrange for glibc to handle them better. IIRC the original problem was not an unfreed 100M but an unfreed *many Gb*. Those are two different problems.