From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: Memory again Date: Sun, 18 Dec 2011 20:34:02 -0500 Message-ID: References: <4ED0F945.5090805@yandex.ru> <83pqge7syw.fsf@gnu.org> <87mxb6tkji.fsf@wanadoo.es> <87borlu0kc.fsf@wanadoo.es> <4EEE0315.60405@yandex.ru> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: dough.gmane.org 1324258455 19763 80.91.229.12 (19 Dec 2011 01:34:15 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Mon, 19 Dec 2011 01:34:15 +0000 (UTC) Cc: emacs-devel@gnu.org To: Dmitry Antipov Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Dec 19 02:34:11 2011 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1RcS7G-0004VF-Ui for ged-emacs-devel@m.gmane.org; Mon, 19 Dec 2011 02:34:11 +0100 Original-Received: from localhost ([::1]:38896 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RcS7G-0004H9-89 for ged-emacs-devel@m.gmane.org; Sun, 18 Dec 2011 20:34:10 -0500 Original-Received: from eggs.gnu.org ([140.186.70.92]:33519) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RcS7D-0004H4-FN for emacs-devel@gnu.org; Sun, 18 Dec 2011 20:34:08 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RcS7C-0007pv-8D for emacs-devel@gnu.org; Sun, 18 Dec 2011 20:34:07 -0500 Original-Received: from smtp-02.vtx.ch ([194.38.175.91]:52054) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RcS7B-0007ph-Nl for emacs-devel@gnu.org; Sun, 18 Dec 2011 20:34:06 -0500 Original-Received: from ceviche.home (dyn.83-228-151-202.dsl.vtx.ch [83.228.151.202]) by smtp-02.vtx.ch (VTX Services SA) with ESMTP id 09B39603FF; Mon, 19 Dec 2011 02:34:03 +0100 (CET) Original-Received: by ceviche.home (Postfix, from userid 20848) id C5C9D66188; Sun, 18 Dec 2011 20:34:02 -0500 (EST) In-Reply-To: <4EEE0315.60405@yandex.ru> (Dmitry Antipov's message of "Sun, 18 Dec 2011 19:13:25 +0400") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.92 (gnu/linux) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Received-From: 194.38.175.91 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:146807 Archived-At: >> - Many memory problems are misdiagnosed as fragmentation problems, where >> the real cause is either inefficient memory use, or memory leaks. >> - Real fragmentation problems exist but are fairly rare. > It would be nice to have a built-in (optionally selected at configure time) > method to provide a 'shot' of the heap to see how it's (mis)used. It's also > interesting whether it's possible to write a set of gdb macros for doing > something similar. `garbage-collect' is supposed to give that info. At least it does give info about the data that's kept under alloc.c's control because of fragmentation (these are counted as "free cells"). >> - In some "fragmentation" cases, the Emacs process holds on to memory >> without any good reason, i.e. it's been "free"d but the malloc library >> does not want or can't return it to the OS because it did not use mmap >> to allocate it. This can be fixed, but this memory would really be >> unused; it can still appear in RSS but only because the memory >> pressure hasn't yet bumped it out of RAM. IOW these cases may show >> high memory use in terms of VSZ or RSS but fixing them is low priority >> because their only direct cost is use of of swap space. > IIUC, this is not true, at least for Linux (see how zap_pte_range() updates > MM_ANONPAGES RSS counter; it's done when munmap() happens). Unmapped > (but still resident in RAM) pages aren't accounted as RSS of any process; > they're accounted separately and amount of space occupied by such pages is > 'Active(anon)' in /proc/meminfo. Re-read what I wrote: I'm talking about the case where alloc.c free'd the data, but malloc did not munmap it. >> - Fixing the remaining real fragmentation problems probably requires >> a different allocator that can move objects to compact the memory >> space. Maybe such an allocator can be retrofitted into Emacs >> (e.g. a mostly-copying allocator), but noone has tried it yet >> (probably because the outcome is hard to predict, the problem it >> attempts to fix only occurs rather rarely, and it can be difficult to >> ensure that it doesn't affect negatively the more common cases). > It's not so hard to predict the memory-saving benefits of copying or > compacting collector - ideally such a collector should free everything It's easy to predict what it does in terms of "what is the benefit when I have a lot of fragmentation". It's much more difficult to predict what it does in terms of "how is it going to affect Emacs's behavior for those 99% cases where it currently works just fine". > hold of a free lists is ~3-5% of total heap size; it's reasonable to expect > that copying/compacting collector may also decrease an overall heap > fragmentation, which may give a few more percents. Anyway, ~5% isn't > convincing enough to start a hard work on a new GC; instead, I believe I see we agree. > that some minor optimization of current algorithm and Lisp data > representation (vector allocation, compact or 'immediate' strings and > something like cdr-coding or unrolled lists) may give comparable, or > even better, effect in the sense of memory consumption. I agree that we're probably going to see better overall results by improving general memory use than by trying to attack fragmentation problems. Stefan