From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Dmitry Antipov Newsgroups: gmane.emacs.devel Subject: Re: Memory again Date: Sun, 18 Dec 2011 19:13:25 +0400 Message-ID: <4EEE0315.60405@yandex.ru> References: <4ED0F945.5090805@yandex.ru> <83pqge7syw.fsf@gnu.org> <87mxb6tkji.fsf@wanadoo.es> <87borlu0kc.fsf@wanadoo.es> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1324221191 19750 80.91.229.12 (18 Dec 2011 15:13:11 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sun, 18 Dec 2011 15:13:11 +0000 (UTC) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Dec 18 16:13:05 2011 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1RcIQ7-0001RQ-RO for ged-emacs-devel@m.gmane.org; Sun, 18 Dec 2011 16:12:59 +0100 Original-Received: from localhost ([::1]:42383 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RcIQ7-0000ZZ-7r for ged-emacs-devel@m.gmane.org; Sun, 18 Dec 2011 10:12:59 -0500 Original-Received: from eggs.gnu.org ([140.186.70.92]:44014) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RcIQ4-0000ZJ-3a for emacs-devel@gnu.org; Sun, 18 Dec 2011 10:12:56 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RcIQ2-0000ja-LN for emacs-devel@gnu.org; Sun, 18 Dec 2011 10:12:56 -0500 Original-Received: from mail.dev.rtsoft.ru ([213.79.90.226]:50258) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1RcIQ2-0000jO-7r for emacs-devel@gnu.org; Sun, 18 Dec 2011 10:12:54 -0500 Original-Received: (qmail 12830 invoked from network); 18 Dec 2011 15:12:50 -0000 Original-Received: from unknown (HELO ?192.168.5.146?) (192.168.1.70) by 0 with SMTP; 18 Dec 2011 15:12:50 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111115 Thunderbird/8.0 In-Reply-To: X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Received-From: 213.79.90.226 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:146799 Archived-At: On 12/17/2011 01:55 AM, Stefan Monnier wrote: > - Many memory problems are misdiagnosed as fragmentation problems, where > the real cause is either inefficient memory use, or memory leaks. > - Real fragmentation problems exist but are fairly rare. It would be nice to have a built-in (optionally selected at configure time) method to provide a 'shot' of the heap to see how it's (mis)used. It's also interesting whether it's possible to write a set of gdb macros for doing something similar. > - In some "fragmentation" cases, the Emacs process holds on to memory > without any good reason, i.e. it's been "free"d but the malloc library > does not want or can't return it to the OS because it did not use mmap > to allocate it. This can be fixed, but this memory would really be > unused; it can still appear in RSS but only because the memory > pressure hasn't yet bumped it out of RAM. IOW these cases may show > high memory use in terms of VSZ or RSS but fixing them is low priority > because their only direct cost is use of of swap space. IIUC, this is not true, at least for Linux (see how zap_pte_range() updates MM_ANONPAGES RSS counter; it's done when munmap() happens). Unmapped (but still resident in RAM) pages aren't accounted as RSS of any process; they're accounted separately and amount of space occupied by such pages is 'Active(anon)' in /proc/meminfo. > - Fixing the remaining real fragmentation problems probably requires > a different allocator that can move objects to compact the memory > space. Maybe such an allocator can be retrofitted into Emacs > (e.g. a mostly-copying allocator), but noone has tried it yet > (probably because the outcome is hard to predict, the problem it > attempts to fix only occurs rather rarely, and it can be difficult to > ensure that it doesn't affect negatively the more common cases). It's not so hard to predict the memory-saving benefits of copying or compacting collector - ideally such a collector should free everything which is on a free lists in current collector, so an output of garbage-collect may be used as a rough estimate of how much data can be compacted. In my measurements, the typical amount of space which is hold of a free lists is ~3-5% of total heap size; it's reasonable to expect that copying/compacting collector may also decrease an overall heap fragmentation, which may give a few more percents. Anyway, ~5% isn't convincing enough to start a hard work on a new GC; instead, I believe that some minor optimization of current algorithm and Lisp data representation (vector allocation, compact or 'immediate' strings and something like cdr-coding or unrolled lists) may give comparable, or even better, effect in the sense of memory consumption. Dmitry