From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Dmitry Antipov Newsgroups: gmane.emacs.devel Subject: Re: Emacs bzr memory footprint Date: Fri, 21 Oct 2011 19:34:41 +0400 Message-ID: <4EA19111.7060401@yandex.ru> References: <83fwix2osa.fsf@gnu.org> <0B3EE7A4-D0D6-4D1E-ADC4-0BEE68F179B2@mit.edu> <87fwivwp37.fsf@turtle.gmx.de> <87sjmvpmd2.fsf@lifelogs.com> <87aa93wmc4.fsf@turtle.gmx.de> <87sjmnrdjw.fsf@spindle.srvr.nix> <87ty73mc0m.fsf@spindle.srvr.nix> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1319211298 17205 80.91.229.12 (21 Oct 2011 15:34:58 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Fri, 21 Oct 2011 15:34:58 +0000 (UTC) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Fri Oct 21 17:34:53 2011 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1RHH7U-0004DP-UV for ged-emacs-devel@m.gmane.org; Fri, 21 Oct 2011 17:34:53 +0200 Original-Received: from localhost ([::1]:51421 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RHH7U-0005SS-Fb for ged-emacs-devel@m.gmane.org; Fri, 21 Oct 2011 11:34:52 -0400 Original-Received: from eggs.gnu.org ([140.186.70.92]:38372) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RHH7R-0005SN-Vw for emacs-devel@gnu.org; Fri, 21 Oct 2011 11:34:51 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RHH7R-0004U7-15 for emacs-devel@gnu.org; Fri, 21 Oct 2011 11:34:49 -0400 Original-Received: from mail.dev.rtsoft.ru ([213.79.90.226]:36109) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1RHH7Q-0004Tx-Km for emacs-devel@gnu.org; Fri, 21 Oct 2011 11:34:48 -0400 Original-Received: (qmail 13895 invoked from network); 21 Oct 2011 15:34:46 -0000 Original-Received: from unknown (HELO ?192.168.5.146?) (192.168.1.70) by 0 with SMTP; 21 Oct 2011 15:34:46 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0) Gecko/20110927 Thunderbird/7.0 In-Reply-To: X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Received-From: 213.79.90.226 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:145400 Archived-At: On 10/21/2011 05:30 PM, Stefan Monnier wrote: > Emacs already uses its own malloc for most objects: it uses malloc > directly to allocate vectors (and this is actually something that we > might want to change, because it (indirectly) comes with an O(log n) > cost), but for strings, conses, markers, overlays, and floats it uses > malloc only to get a chunk of memory which it then manages on its own. > `memory-usage' does give you this kind of info when it says > "5743392+864696 bytes in cons cells" which means 5MB of live cons-cells, > and 800KB of cons-cells that are free (i.e. they are in a memory chunk > that Emacs can't return to malloc because that chunk also contains live > cons cells). This creates two-level fragmentation: the whole heap is fragmented as seen by malloc(), and allocated space is fragmented too, as seen by Emacs. Since a lot of non-lisp allocation requests are satisfied from the same sbrk()ed heap, here is the typical scenario: at startup, there is 10M of live lisp data. Solid workload (e.g. visit 1000 files at once) increases this, say, to 100M. When the workload is finished (kill all file-backed 1000 buffers), live lisp data shrinks back to 10M. The rest 90M can't be freed because a) small percentage is interleaved with live lisp data within the chunks managed by Emacs itself and b) the rest, being freed by Emacs, can't be freed by malloc because freed chunks are interleaved with non-freed, and all chunks are interleaved with freed/lived lisp vectors and freed/lived non-lisp allocations. If next solid workload will consume 100M of the heap, the most of this 90M will be re-used - but it will never freed. Due to all of the above, I suspect that the mostly-copying collector, where all small (say, < 4K) lisp (and only lisp) allocation requests are satisfied from 1M-4M areas, might have substantial advantages over plain mark&sweep scheme. Dmitry