From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: Proposal: block-based vector allocator Date: Thu, 08 Dec 2011 14:58:04 -0500 Message-ID: References: <4EDDA68B.5050601@yandex.ru> <4EDE315E.3030804@yandex.ru> <4EDEF421.6090800@yandex.ru> <4EDF8F8E.8050401@yandex.ru> <87hb1b25i7.fsf@uwakimon.sk.tsukuba.ac.jp> <4EE03FFD.1090400@yandex.ru> <4EE0EA5C.1090505@yandex.ru> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: dough.gmane.org 1323374294 25621 80.91.229.12 (8 Dec 2011 19:58:14 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Thu, 8 Dec 2011 19:58:14 +0000 (UTC) Cc: emacs-devel@gnu.org To: Dmitry Antipov Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Thu Dec 08 20:58:10 2011 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1RYk6c-0004AU-B8 for ged-emacs-devel@m.gmane.org; Thu, 08 Dec 2011 20:58:10 +0100 Original-Received: from localhost ([::1]:45316 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RYk6b-0007Qx-LG for ged-emacs-devel@m.gmane.org; Thu, 08 Dec 2011 14:58:09 -0500 Original-Received: from eggs.gnu.org ([140.186.70.92]:50749) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RYk6a-0007Qs-3j for emacs-devel@gnu.org; Thu, 08 Dec 2011 14:58:08 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RYk6Y-0000tY-Ov for emacs-devel@gnu.org; Thu, 08 Dec 2011 14:58:08 -0500 Original-Received: from pruche.dit.umontreal.ca ([132.204.246.22]:43879) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RYk6Y-0000tU-Ei for emacs-devel@gnu.org; Thu, 08 Dec 2011 14:58:06 -0500 Original-Received: from pastel.home (lechon.iro.umontreal.ca [132.204.27.242]) by pruche.dit.umontreal.ca (8.14.1/8.14.1) with ESMTP id pB8Jw4bA005925; Thu, 8 Dec 2011 14:58:05 -0500 Original-Received: by pastel.home (Postfix, from userid 20848) id 35A325938C; Thu, 8 Dec 2011 14:58:04 -0500 (EST) In-Reply-To: <4EE0EA5C.1090505@yandex.ru> (Dmitry Antipov's message of "Thu, 08 Dec 2011 20:48:28 +0400") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.91 (gnu/linux) X-NAI-Spam-Flag: NO X-NAI-Spam-Threshold: 5 X-NAI-Spam-Score: 0 X-NAI-Spam-Rules: 1 Rules triggered RV4065=0 X-NAI-Spam-Version: 2.2.0.9309 : core <4065> : streams <708915> : uri <1024201> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 132.204.246.22 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:146586 Archived-At: >> Someone could try to change the allocator so those blocks are exactly >> 1-page in size and get allocated via mmap: > I did this even before I started with block-based vector allocator. How much did it help? >> - using mmap will ensure that we factor out malloc from the equation. >> - using page-size (presumably 4KB) blocks rather than the current 16KB > Current sizes are 1K except string data, which is 8K. For cons cells and floats, we call malloc for 16KB blocks at a time (which we then cut into 1024 blocks). This is because we need the 1024 blocks to be aligned on a 1024 boundary (so when we use malloc instead of posix_memalign, we waste upto 1023 (well 1016, really) bytes of those 16KB). > My favorite byte-force-recompile benchmark usually consumes ~89M. With my > vector allocator it shrinks to 78M. And you said that using the non-conservative stack scanning resulted in similar results, so all that extra memory is occupied by mem_nodes (or fragments that used to hole mem_nodes ;-). > But for (definitely pathological) compile buffer case, results are > very unexpected (I tried with 100000-lines buffer). Fresh instance > is 18M. With compile buffer, it grows to 142M and shrinks to 141M > after killing *compilation* and garbage-collect. When mmap() is enabled, > it grows to 138M and shrinks back to 21M (twenty one) after kill & GC. Hmm.. cool. So some of the responsability for this test to behave so poorly might be in malloc rather than in alloc.c. Good news. Oh, yes, now I remember: #ifdef DOUG_LEA_MALLOC /* Prevent mmap'ing the chunk. Lisp data may not be mmap'ed because mapped region contents are not preserved in a dumped Emacs. */ mallopt (M_MMAP_MAX, 0); #endif So maybe malloc behaves poorly simply because we don't let it behave better. Stefan