From: Dmitry Antipov <dmantipov@yandex.ru>
To: emacs-devel@gnu.org
Subject: Re: Memory again
Date: Sun, 18 Dec 2011 19:13:25 +0400 [thread overview]
Message-ID: <4EEE0315.60405@yandex.ru> (raw)
In-Reply-To: <jwvmxasb3rp.fsf-monnier+emacs@gnu.org>
On 12/17/2011 01:55 AM, Stefan Monnier wrote:
> - Many memory problems are misdiagnosed as fragmentation problems, where
> the real cause is either inefficient memory use, or memory leaks.
> - Real fragmentation problems exist but are fairly rare.
It would be nice to have a built-in (optionally selected at configure time)
method to provide a 'shot' of the heap to see how it's (mis)used. It's also
interesting whether it's possible to write a set of gdb macros for doing
something similar.
> - In some "fragmentation" cases, the Emacs process holds on to memory
> without any good reason, i.e. it's been "free"d but the malloc library
> does not want or can't return it to the OS because it did not use mmap
> to allocate it. This can be fixed, but this memory would really be
> unused; it can still appear in RSS but only because the memory
> pressure hasn't yet bumped it out of RAM. IOW these cases may show
> high memory use in terms of VSZ or RSS but fixing them is low priority
> because their only direct cost is use of of swap space.
IIUC, this is not true, at least for Linux (see how zap_pte_range() updates
MM_ANONPAGES RSS counter; it's done when munmap() happens). Unmapped
(but still resident in RAM) pages aren't accounted as RSS of any process;
they're accounted separately and amount of space occupied by such pages is
'Active(anon)' in /proc/meminfo.
> - Fixing the remaining real fragmentation problems probably requires
> a different allocator that can move objects to compact the memory
> space. Maybe such an allocator can be retrofitted into Emacs
> (e.g. a mostly-copying allocator), but noone has tried it yet
> (probably because the outcome is hard to predict, the problem it
> attempts to fix only occurs rather rarely, and it can be difficult to
> ensure that it doesn't affect negatively the more common cases).
It's not so hard to predict the memory-saving benefits of copying or
compacting collector - ideally such a collector should free everything
which is on a free lists in current collector, so an output of
garbage-collect may be used as a rough estimate of how much data can
be compacted. In my measurements, the typical amount of space which is
hold of a free lists is ~3-5% of total heap size; it's reasonable to expect
that copying/compacting collector may also decrease an overall heap
fragmentation, which may give a few more percents. Anyway, ~5% isn't
convincing enough to start a hard work on a new GC; instead, I believe
that some minor optimization of current algorithm and Lisp data representation
(vector allocation, compact or 'immediate' strings and something like cdr-coding
or unrolled lists) may give comparable, or even better, effect in the sense
of memory consumption.
Dmitry
next prev parent reply other threads:[~2011-12-18 15:13 UTC|newest]
Thread overview: 87+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-11-26 13:26 Memory again Carsten Mattner
2011-11-26 13:28 ` Carsten Mattner
2011-11-26 14:35 ` Dmitry Antipov
2011-11-26 14:48 ` Eli Zaretskii
2011-11-26 17:37 ` Dmitry Antipov
2011-11-26 20:19 ` Eli Zaretskii
2011-11-26 14:58 ` Carsten Mattner
2011-11-26 16:23 ` Eli Zaretskii
2011-11-26 19:02 ` Carsten Mattner
2011-11-26 20:31 ` Eli Zaretskii
2011-11-26 21:00 ` Eli Zaretskii
2011-11-27 10:29 ` Carsten Mattner
2011-11-27 10:43 ` Andreas Schwab
2011-11-27 13:53 ` Carsten Mattner
2011-11-27 13:11 ` Eli Zaretskii
2011-11-27 13:53 ` Carsten Mattner
2011-11-27 16:44 ` Eli Zaretskii
2011-11-27 17:37 ` Carsten Mattner
2011-11-27 17:59 ` Carsten Mattner
2011-12-06 4:02 ` Óscar Fuentes
2011-12-06 5:08 ` Eli Zaretskii
2011-12-06 9:35 ` Carsten Mattner
2011-12-06 10:24 ` Dmitry Antipov
2011-12-06 13:07 ` Eli Zaretskii
2011-12-06 13:29 ` Stefan Monnier
2011-12-06 17:20 ` Eli Zaretskii
2011-12-06 20:25 ` Stefan Monnier
2011-12-07 7:52 ` Eli Zaretskii
2011-12-07 8:15 ` Dmitry Antipov
2011-12-07 13:06 ` Eli Zaretskii
2011-12-07 14:01 ` Stefan Monnier
2011-12-08 17:30 ` Carsten Mattner
2011-12-09 3:39 ` Dmitry Antipov
2011-12-09 13:52 ` Carsten Mattner
2011-12-06 13:12 ` Eli Zaretskii
2011-12-06 16:28 ` Óscar Fuentes
2011-12-06 19:53 ` Stefan Monnier
2011-12-11 17:49 ` Nix
2011-12-15 3:52 ` Tim Connors
2011-12-15 4:09 ` Eli Zaretskii
2011-12-15 4:38 ` Tim Connors
2011-12-15 5:52 ` Eli Zaretskii
2011-12-15 4:50 ` Óscar Fuentes
2011-12-15 6:04 ` Eli Zaretskii
2011-12-16 21:55 ` Stefan Monnier
2011-12-17 17:40 ` Nix
2011-12-18 15:13 ` Dmitry Antipov [this message]
2011-12-19 1:34 ` Stefan Monnier
2011-12-19 8:28 ` Dmitry Antipov
2011-12-19 11:26 ` Stefan Monnier
2012-01-23 16:49 ` Nix
2012-01-25 16:19 ` Ted Zlatanov
2011-11-26 17:54 ` Dmitry Antipov
2011-11-26 18:47 ` martin rudalics
2011-11-26 19:09 ` Carsten Mattner
2011-11-28 4:27 ` Stefan Monnier
2011-11-28 9:24 ` Carsten Mattner
2011-11-28 15:31 ` Davis Herring
2011-11-28 21:33 ` Carsten Mattner
-- strict thread matches above, loose matches on Subject: below --
2011-12-19 19:51 emacs user
2011-12-20 5:32 ` Dmitry Antipov
2012-01-06 14:28 ` Chong Yidong
2012-01-06 15:53 ` emacs user
2011-12-20 6:34 emacs user
2011-12-20 7:43 ` Eli Zaretskii
2011-12-20 12:05 ` emacs user
2011-12-20 13:04 ` Eli Zaretskii
2011-12-20 22:07 ` Jan Djärv
2011-12-21 8:07 ` Jan Djärv
2011-12-21 10:39 ` Carsten Mattner
2011-12-21 17:55 ` emacs user
2011-12-22 14:08 ` Jan Djärv
2011-12-22 14:58 ` emacs user
2011-12-22 18:54 ` emacs user
2011-12-22 19:15 ` Jan Djärv
2011-12-23 4:41 ` YAMAMOTO Mitsuharu
2012-01-17 10:04 ` emacs user
2012-01-17 10:58 ` YAMAMOTO Mitsuharu
2012-01-17 13:14 ` emacs user
2012-01-18 1:30 ` YAMAMOTO Mitsuharu
2011-12-22 23:09 ` Carsten Mattner
2011-12-23 0:39 ` Stefan Monnier
2011-12-23 10:44 ` emacs user
2012-01-05 6:13 ` emacs user
2012-01-05 22:37 ` Jan Djärv
2012-01-06 9:58 ` emacs user
2012-01-06 11:10 ` Carsten Mattner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/emacs/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4EEE0315.60405@yandex.ru \
--to=dmantipov@yandex.ru \
--cc=emacs-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).