From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Dmitry Antipov Newsgroups: gmane.emacs.devel Subject: Re: Memory again Date: Mon, 19 Dec 2011 12:28:36 +0400 Message-ID: <4EEEF5B4.3050806@yandex.ru> References: <4ED0F945.5090805@yandex.ru> <83pqge7syw.fsf@gnu.org> <87mxb6tkji.fsf@wanadoo.es> <87borlu0kc.fsf@wanadoo.es> <4EEE0315.60405@yandex.ru> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------090500090805060204080001" X-Trace: dough.gmane.org 1324283295 23972 80.91.229.12 (19 Dec 2011 08:28:15 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Mon, 19 Dec 2011 08:28:15 +0000 (UTC) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Dec 19 09:28:10 2011 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1RcYZs-0008Q6-5r for ged-emacs-devel@m.gmane.org; Mon, 19 Dec 2011 09:28:08 +0100 Original-Received: from localhost ([::1]:36505 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RcYZq-0004ez-Ra for ged-emacs-devel@m.gmane.org; Mon, 19 Dec 2011 03:28:06 -0500 Original-Received: from eggs.gnu.org ([140.186.70.92]:43592) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RcYZo-0004eu-LH for emacs-devel@gnu.org; Mon, 19 Dec 2011 03:28:05 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RcYZn-0000Gk-AV for emacs-devel@gnu.org; Mon, 19 Dec 2011 03:28:04 -0500 Original-Received: from mail.dev.rtsoft.ru ([213.79.90.226]:58568) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1RcYZm-0000GY-KO for emacs-devel@gnu.org; Mon, 19 Dec 2011 03:28:03 -0500 Original-Received: (qmail 21496 invoked from network); 19 Dec 2011 08:28:00 -0000 Original-Received: from unknown (HELO ?192.168.5.146?) (192.168.1.70) by 0 with SMTP; 19 Dec 2011 08:28:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111115 Thunderbird/8.0 In-Reply-To: X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Received-From: 213.79.90.226 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:146811 Archived-At: This is a multi-part message in MIME format. --------------090500090805060204080001 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 12/19/2011 05:34 AM, Stefan Monnier wrote: > `garbage-collect' is supposed to give that info. At least it does give > info about the data that's kept under alloc.c's control because > of fragmentation (these are counted as "free cells"). I would like to propose a function which explicitly says how much free memory the Emacs process holds. It's especially useful when there is a way to ask system malloc about how much free memory is in the heap. This may be helpful to distinguish real heap fragmentation from memory leaks and other misuses - if the sum of values returned by 'memory-free' is, say, 10% of heap size, then the real fragmentation enters into the game. > Re-read what I wrote: I'm talking about the case where alloc.c free'd the > data, but malloc did not munmap it. I read it as 'even if we will fix allocation to use mmap and munmap, unmapped memory would really be unused and still appear in process RSS, thus giving higher RSS value than the process really has, until the memory pressure bump it out of RAM'. That was obviously incorrect because RSS is recalculated each time when mmap/mremap/munmap/brk is (successfully) called. > It's easy to predict what it does in terms of "what is the benefit when > I have a lot of fragmentation". It's much more difficult to predict > what it does in terms of "how is it going to affect Emacs's behavior for > those 99% cases where it currently works just fine". There is an interesting research paper about moving PLT Scheme from mark-sweep GC to an accurate copying GC: http://www.cs.utah.edu/plt/publications/ismm09-rwrf.pdf. In short, it's said about getting rid out of the most fragmentation issues at the cost of 10-20% slowdown (although I consider their approach as practically inapplicable beyond the pure research efforts). > I agree that we're probably going to see better overall results by > improving general memory use than by trying to attack > fragmentation problems. Among this list's subscribers, Nix is constantly reporting an enormous memory usage caused by Gnus. Since I newer seen such a heap usage in my own test cases, I offered him a 'cumulative' patch (against 24.0.92) with both immediate strings and block vector allocator. He has said that we can expect the first results after Christmas :-). Dmitry --------------090500090805060204080001 Content-Type: text/plain; name="memory_free.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="memory_free.patch" PT09IG1vZGlmaWVkIGZpbGUgJ3NyYy9hbGxvYy5jJwotLS0gc3JjL2FsbG9jLmMJMjAxMS0x Mi0xMiAwNTozMjo0OSArMDAwMAorKysgc3JjL2FsbG9jLmMJMjAxMS0xMi0xOSAwNjowNjow NCArMDAwMApAQCAtNzUsNiArNzUsMTAgQEAKIAogI2RlZmluZSBNTUFQX01BWF9BUkVBUyAx MDAwMDAwMDAKIAorLyogVmFsdWUgb2YgbWFsbGluZm8gKCkuZm9yZGJsa3MgYXMgc2VlbiBh dCB0aGUgZW5kIG9mIGxhc3QgR0MuICAqLworCitzdGF0aWMgaW50IGJ5dGVzX2ZyZWU7CisK ICNlbHNlIC8qIG5vdCBET1VHX0xFQV9NQUxMT0MgKi8KIAogLyogVGhlIGZvbGxvd2luZyBj b21lIGZyb20gZ21hbGxvYy5jLiAgKi8KQEAgLTUyNzEsNiArNTI3NSwxMCBAQAogICB0b3Rh bFs3XSA9IEZjb25zIChtYWtlX251bWJlciAodG90YWxfc3RyaW5ncyksCiAJCSAgICBtYWtl X251bWJlciAodG90YWxfZnJlZV9zdHJpbmdzKSk7CiAKKyNpZmRlZiBET1VHX0xFQV9NQUxM T0MKKyAgYnl0ZXNfZnJlZSA9IG1hbGxpbmZvICgpLmZvcmRibGtzOworI2VuZGlmCisKICNp ZiBHQ19NQVJLX1NUQUNLID09IEdDX1VTRV9HQ1BST1NfQ0hFQ0tfWk9NQklFUwogICB7CiAg ICAgLyogQ29tcHV0ZSBhdmVyYWdlIHBlcmNlbnRhZ2Ugb2Ygem9tYmllcy4gICovCkBAIC02 MjIxLDYgKzYyMjksMzMgQEAKICAgcmV0dXJuIGVuZDsKIH0KIAorREVGVU4gKCJtZW1vcnkt ZnJlZSIsIEZtZW1vcnlfZnJlZSwgU21lbW9yeV9mcmVlLCAwLCAwLCAwLAorICAgICAgIGRv YzogLyogUmV0dXJuIGEgbGlzdCBvZiB0d28gY291bnRlcnMgdGhhdCBtZWFzdXJlIGhvdyBt dWNoIGZyZWUKK21lbW9yeSBpcyBob2xkIGJ5IHRoZSBFbWFjcyBwcm9jZXNzLiBCb3RoIGNv dW50ZXJzIGFyZQoraW4gS0J5dGVzLiBGaXJzdCBjb3VudGVyIHNob3dzIGhvdyBtdWNoIG1l bW9yeSBob2xkcyBpbgorYSBmcmVlIGxpc3RzIG1haW50YWluZWQgYnkgdGhlIEVtYWNzIGl0 c2VsZi4gU2Vjb25kIGNvdW50ZXIKK3Nob3dzIGhvdyBtdWNoIGZyZWUgbWVtb3J5IGlzIGlu IHRoZSBoZWFwLiBJZiB0aGUgc2Vjb25kCitjb3VudGVyIGlzIHplcm8sIGhlYXAgc3RhdGlz dGljcyBpcyBub3QgYXZhaWxhYmxlLiAgKi8pCisgICAgICh2b2lkKQoreworICBMaXNwX09i amVjdCBkYXRhWzJdOworICAKKyAgZGF0YVswXSA9IG1ha2VfbnVtYmVyCisgICAgKG1pbiAo TU9TVF9QT1NJVElWRV9GSVhOVU0sCisJICAodG90YWxfZnJlZV9jb25zZXMgKiBzaXplb2Yg KHN0cnVjdCBMaXNwX0NvbnMpICsKKwkgICB0b3RhbF9mcmVlX21hcmtlcnMgKiBzaXplb2Yg KHVuaW9uIExpc3BfTWlzYykgKworCSAgIHRvdGFsX2ZyZWVfc3ltYm9scyAqIHNpemVvZiAo c3RydWN0IExpc3BfU3ltYm9sKSArCisJICAgdG90YWxfZnJlZV9mbG9hdHMgKiBzaXplb2Yg KHN0cnVjdCBMaXNwX0Zsb2F0KSArCisJICAgdG90YWxfZnJlZV9pbnRlcnZhbHMgKiBzaXpl b2YgKHN0cnVjdCBpbnRlcnZhbCkgKworCSAgIHRvdGFsX2ZyZWVfc3RyaW5ncyAqIHNpemVv ZiAoc3RydWN0IExpc3BfU3RyaW5nKSkgLyAxMDI0KSk7CisjaWZkZWYgRE9VR19MRUFfTUFM TE9DCisgIGRhdGFbMV0gPSBtYWtlX251bWJlciAobWluIChNT1NUX1BPU0lUSVZFX0ZJWE5V TSwgYnl0ZXNfZnJlZSAvIDEwMjQpKTsKKyNlbHNlCisgIGRhdGFbMV0gPSBtYWtlX251bWJl ciAoMCk7CisjZW5kaWYKKyAgcmV0dXJuIEZsaXN0ICgyLCBkYXRhKTsKK30KKwogREVGVU4g KCJtZW1vcnktdXNlLWNvdW50cyIsIEZtZW1vcnlfdXNlX2NvdW50cywgU21lbW9yeV91c2Vf Y291bnRzLCAwLCAwLCAwLAogICAgICAgIGRvYzogLyogUmV0dXJuIGEgbGlzdCBvZiBjb3Vu dGVycyB0aGF0IG1lYXN1cmUgaG93IG11Y2ggY29uc2luZyB0aGVyZSBoYXMgYmVlbi4KIEVh Y2ggb2YgdGhlc2UgY291bnRlcnMgaW5jcmVtZW50cyBmb3IgYSBjZXJ0YWluIGtpbmQgb2Yg b2JqZWN0LgpAQCAtNjQ3NCw2ICs2NTA5LDcgQEAKICAgZGVmc3ViciAoJlNwdXJlY29weSk7 CiAgIGRlZnN1YnIgKCZTZ2FyYmFnZV9jb2xsZWN0KTsKICAgZGVmc3ViciAoJlNtZW1vcnlf bGltaXQpOworICBkZWZzdWJyICgmU21lbW9yeV9mcmVlKTsKICAgZGVmc3ViciAoJlNtZW1v cnlfdXNlX2NvdW50cyk7CiAKICNpZiBHQ19NQVJLX1NUQUNLID09IEdDX1VTRV9HQ1BST1Nf Q0hFQ0tfWk9NQklFUwoK --------------090500090805060204080001--