From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.bugs Subject: bug#45200: [PATCH] Force Glibc to free the memory freed Date: Wed, 03 Feb 2021 18:32:36 -0500 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="3075"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: fweimer@redhat.com, carlos@redhat.com, hi-angel@yandex.ru, 45200@debbugs.gnu.org To: DJ Delorie Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Feb 04 00:33:30 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l7ReM-0000iQ-6l for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 04 Feb 2021 00:33:30 +0100 Original-Received: from localhost ([::1]:38572 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7ReL-0004gB-7g for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 03 Feb 2021 18:33:29 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:44620) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Rdu-0004fj-Gn for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 18:33:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:55862) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l7Rdu-0000FR-9Y for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 18:33:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l7Rdu-0001fJ-6a for bug-gnu-emacs@gnu.org; Wed, 03 Feb 2021 18:33:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 03 Feb 2021 23:33:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45200 X-GNU-PR-Package: emacs Original-Received: via spool by 45200-submit@debbugs.gnu.org id=B45200.16123951806393 (code B ref 45200); Wed, 03 Feb 2021 23:33:02 +0000 Original-Received: (at 45200) by debbugs.gnu.org; 3 Feb 2021 23:33:00 +0000 Original-Received: from localhost ([127.0.0.1]:39175 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l7Rdr-0001f3-Ka for submit@debbugs.gnu.org; Wed, 03 Feb 2021 18:32:59 -0500 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:21591) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l7Rdq-0001eo-Cf for 45200@debbugs.gnu.org; Wed, 03 Feb 2021 18:32:58 -0500 Original-Received: from pmg2.iro.umontreal.ca (localhost.localdomain [127.0.0.1]) by pmg2.iro.umontreal.ca (Proxmox) with ESMTP id C781F80976; Wed, 3 Feb 2021 18:32:52 -0500 (EST) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg2.iro.umontreal.ca (Proxmox) with ESMTP id 240F78063A; Wed, 3 Feb 2021 18:32:38 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1612395158; bh=+Lk1WLoj32NkCn77sEIZxJ7opV7/J9YxMoSzj79fmow=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=CqwlJdS4bwr82nEfk2OvbbhtXJaaAoj/JbZPWiHhMPt12jF1GqqWKgSw/57KIyH6+ a/mLCzWJXfRtbhCAvaSUwugaE3W/4bXM1d/bdNMVSkVcRT+skTe8c3B/6o4LOerwVW ffwlESGlOaqw7RJUsopwElwkl/9MJsfoJCvn3ue6xU+KeLRJM4TZYVlS7YREEP4GkC fGRcuJV6QpaohM9I/caieNRta7iu1ltywF9aRWF5QRBtZOC4bG+KLD8d9Q6wk/gxSz TXZE5NTZRcH/LeHl8tsArynJSJLm9CD/mWOKciqOGp/vEBkLlnVJ2bD/a4fLRYw2M/ 1+hnJV3QRhc1g== Original-Received: from alfajor (76-10-182-85.dsl.teksavvy.com [76.10.182.85]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id B97A012020B; Wed, 3 Feb 2021 18:32:37 -0500 (EST) In-Reply-To: (DJ Delorie's message of "Wed, 03 Feb 2021 17:21:44 -0500") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:199254 Archived-At: >> - one `static unsigned long hoard_size` keeps the approximate amount of >> space that is free but not returned to the OS. >> Not sure where/when to keep it up to date cheaply, admittedly. > With full atomic access because malloc is multi-threaded; atomics are > expensive compared to thread-local or nonexisting data. Updating this > has to be in the critical path, too. Indeed, multithreading makes it all much more fun. I guess you'd want to make those info thread-local, which would force a rethink of its definition (what about data allocated in one thread bu freed in another, yadda, yadda). The key would be to make sure this is not updated for every call to `free` but only for some of them (that's why it says "approximate"). Not having the source code of `free` at hand, I'm free to assume that this is possible. E.g. we could say that we don't count objects smaller than a certain size or for objects smaller than 2^N (for some arbitrary 2^N), you only count it if it's placed at the beginning of the 2^N chunk (and you count it as having size 2^N)? >> - one `static unsigned long smallest_recent_hoard_size`. >> This is updated whenever we allocate memory from the OS. > We map huge chunks at a time, but they stay unbacked until they're > actually used. By "whenever we allocate memory from the OS" I really meant "in a code path that's not speed critical but which can be expected to be executed eventually". Maybe there isn't any such path except for those that may never be executed? >> Then you'd call `malloc_trim` based on a magic formula > That's cheating ;-) Like the rest of my hand waving wasn't? C'mon, give me some credit! [ Just to clarify: by "magic formula" I meant that this would be the part amenable to tuning to decide how often we're willing to pay for `malloc_trim`. ] >>> Yet another word of memory used, >> Since 200MB is peanuts, I figure that extra 24B should be acceptable ;-) > *per chunk*. No, those counters are meant to be global. Anyway. I see that we should presume that our glibc will not release our memory back to the OS unless we explicitly ask it to with `malloc_trim`. Thanks. One thing that remains a bit unclear for me is in the doc of `malloc_trim`; it says: The pad argument specifies the amount of free space to leave untrimmed at the top of the heap. If this argument is 0, only the minimum amount of memory is maintained at the top of the heap (i.e., one page or less). A nonzero argument can be used to maintain some trailing space at the top of the heap in order to allow future allocations to be made without having to extend the heap with sbrk(2). But this only talks about the free space at the "top". Since in our case most of the free space is not at the top, I wonder: say we have 64KB free at the top and 64MB free elsewhere and we call `malloc_trim` with a `pad` of 16MB, will it release ~48MB of the non-top free memory or will it free all 64MB of the non-top free memory or ... ? Stefan