From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.bugs Subject: bug#45200: [PATCH] Force Glibc to free the memory freed Date: Wed, 03 Feb 2021 15:24:23 -0500 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36969"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: fweimer@redhat.com, carlos@redhat.com, hi-angel@yandex.ru, 45200@debbugs.gnu.org To: DJ Delorie Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Feb 03 21:25:44 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l7Oie-0009XX-DD for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 03 Feb 2021 21:25:44 +0100 Original-Received: from localhost ([::1]:43830 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Oic-0000zf-KF for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 03 Feb 2021 15:25:43 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:60588) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Ohy-0000xl-ET for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 15:25:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:55639) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l7Ohy-00037X-5o for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 15:25:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l7Ohx-0003RW-NY for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 15:25:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 03 Feb 2021 20:25:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45200 X-GNU-PR-Package: emacs Original-Received: via spool by 45200-submit@debbugs.gnu.org id=B45200.161238387313193 (code B ref 45200); Wed, 03 Feb 2021 20:25:01 +0000 Original-Received: (at 45200) by debbugs.gnu.org; 3 Feb 2021 20:24:33 +0000 Original-Received: from localhost ([127.0.0.1]:38952 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l7OhU-0003Qj-TY for submit@debbugs.gnu.org; Wed, 03 Feb 2021 15:24:33 -0500 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:47010) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l7OhT-0003QX-Cx for 45200@debbugs.gnu.org; Wed, 03 Feb 2021 15:24:31 -0500 Original-Received: from pmg3.iro.umontreal.ca (localhost [127.0.0.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id 1810544098C; Wed, 3 Feb 2021 15:24:26 -0500 (EST) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id 8A004440974; Wed, 3 Feb 2021 15:24:24 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1612383864; bh=WOpYATkMILoG7l2Y5EFF721MOq3ssNrf2gUqe0bjLjw=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=OqKl1R/K7q0piTC4p5L32/+JxuVceg97fZquRbdJ0wDS8tKg+BMwTEi03GQ0kXbCh /TcXN+R8EppDZwN4ryLjid64d5YvuF/CmWk0klisiZ5syQ3VUg2sagv4CsttXiV4Ov cJoR+rUsHr2U630jORn4xDao3JC8om0eRdzYQfAo77EYKbpYbCvcS1lTIfc3xjCuDs dTJ4sCW8nFEOXBEfRo75hGhdeeokZBVVfuIb8Goq6LvH72fgTN8gHxsREDc2rSAAix c5PHGPB54KZu5VzMOeWWbRpYnVP2agMwOFtJy15Xn2j/HzfUyR8hAA1yL/lj0bfrub iF2CR0CxE+SUg== Original-Received: from alfajor (76-10-182-85.dsl.teksavvy.com [76.10.182.85]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 3B077120218; Wed, 3 Feb 2021 15:24:24 -0500 (EST) In-Reply-To: (DJ Delorie's message of "Wed, 03 Feb 2021 14:25:15 -0500") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:199240 Archived-At: DJ Delorie [2021-02-03 14:25:15] wrote: > Right, glibc's malloc maintains a cache of re-usable chunks of memory, > for performance reasons. Right, so far as good. > We (glibc devs) don't consider "poor cache performance for my > application" to be a bug. A patch which improves cache performance *for > most apps* would be considered, but a patch that improves one app's > speed at the cost of most other apps, would be rejected. I understand that as well. But I'm wondering why glibc is willing to keep *indefinitely* an unused 200MB of memory, which is more than double the mount of memory in use for the rest of the application's life. I mean I understand that you can't predict the future, but I expected that "at some point" glibc should decide that those 200MB have been left unused for long enough that they deserve to be returned to the OS. The doc of `malloc_trim` suggests it's occasionally called by `free` and `mallopt` suggests via `M_TRIM_THRESHOLD` that there's a limit to how much extra spare memory glibc keeps around, so this suggests that indeed memory is trimmed "every once in a while". So what causes Emacs's experience to be different? If I read `mallopt`s man page carefully I see it say: M_TRIM_THRESHOLD When the amount of contiguous free memory at the top of the heap grows sufficiently large, free(3) employs sbrk(2) to release this memory back to the system. (This can be useful in programs so my guess is that in Emacs's case those 200MB of extra memory are *not* contiguous? Would that be it? If so, it does seem like a weakness of glibc's heuristic. Is there a way for Emacs to ask malloc how much memory (or some approximation of it) would be recovered by `malloc_trim`? > You (emacs devs) need to decide whether you care more about memory > footprint or memory performance, and tune malloc accordingly. > malloc_trim() is one knob to tune, there are others. But don't say it's > a "bug" if our defaults don't match your optimal settings. In our normal use, glibc's tuning works fine. We're just seeing some corner case behaviors which seem clearly undesirable, so I think it would make sense to try and arrange for glibc to handle them better. I suspect the issue is how to make it so the malloc library can keep track of the size of the "cache" without slowing down the most common execution path. Once that is done, it should be easy to check the size of the "cache" every once in a while (e.g. when requesting more memory from the OS) and call malloc_trim when that cache is "way too big". Tho of course, "way too big" might just mean we're just about to re-allocate it, so we'd probably want to include some notion of time so we only call `malloc_trim` if it's been "way too big" for a while. Stefan