From: Stefan Monnier <monnier@iro.umontreal.ca>
To: Dmitry Antipov <dmantipov@yandex.ru>
Cc: emacs-devel@gnu.org
Subject: Re: Memory again
Date: Sun, 18 Dec 2011 20:34:02 -0500 [thread overview]
Message-ID: <jwvaa6p8i5i.fsf-monnier+emacs@gnu.org> (raw)
In-Reply-To: <4EEE0315.60405@yandex.ru> (Dmitry Antipov's message of "Sun, 18 Dec 2011 19:13:25 +0400")
>> - Many memory problems are misdiagnosed as fragmentation problems, where
>> the real cause is either inefficient memory use, or memory leaks.
>> - Real fragmentation problems exist but are fairly rare.
> It would be nice to have a built-in (optionally selected at configure time)
> method to provide a 'shot' of the heap to see how it's (mis)used. It's also
> interesting whether it's possible to write a set of gdb macros for doing
> something similar.
`garbage-collect' is supposed to give that info. At least it does give
info about the data that's kept under alloc.c's control because
of fragmentation (these are counted as "free cells").
>> - In some "fragmentation" cases, the Emacs process holds on to memory
>> without any good reason, i.e. it's been "free"d but the malloc library
>> does not want or can't return it to the OS because it did not use mmap
>> to allocate it. This can be fixed, but this memory would really be
>> unused; it can still appear in RSS but only because the memory
>> pressure hasn't yet bumped it out of RAM. IOW these cases may show
>> high memory use in terms of VSZ or RSS but fixing them is low priority
>> because their only direct cost is use of of swap space.
> IIUC, this is not true, at least for Linux (see how zap_pte_range() updates
> MM_ANONPAGES RSS counter; it's done when munmap() happens). Unmapped
> (but still resident in RAM) pages aren't accounted as RSS of any process;
> they're accounted separately and amount of space occupied by such pages is
> 'Active(anon)' in /proc/meminfo.
Re-read what I wrote: I'm talking about the case where alloc.c free'd the
data, but malloc did not munmap it.
>> - Fixing the remaining real fragmentation problems probably requires
>> a different allocator that can move objects to compact the memory
>> space. Maybe such an allocator can be retrofitted into Emacs
>> (e.g. a mostly-copying allocator), but noone has tried it yet
>> (probably because the outcome is hard to predict, the problem it
>> attempts to fix only occurs rather rarely, and it can be difficult to
>> ensure that it doesn't affect negatively the more common cases).
> It's not so hard to predict the memory-saving benefits of copying or
> compacting collector - ideally such a collector should free everything
It's easy to predict what it does in terms of "what is the benefit when
I have a lot of fragmentation". It's much more difficult to predict
what it does in terms of "how is it going to affect Emacs's behavior for
those 99% cases where it currently works just fine".
> hold of a free lists is ~3-5% of total heap size; it's reasonable to expect
> that copying/compacting collector may also decrease an overall heap
> fragmentation, which may give a few more percents. Anyway, ~5% isn't
> convincing enough to start a hard work on a new GC; instead, I believe
I see we agree.
> that some minor optimization of current algorithm and Lisp data
> representation (vector allocation, compact or 'immediate' strings and
> something like cdr-coding or unrolled lists) may give comparable, or
> even better, effect in the sense of memory consumption.
I agree that we're probably going to see better overall results by
improving general memory use than by trying to attack
fragmentation problems.
Stefan
next prev parent reply other threads:[~2011-12-19 1:34 UTC|newest]
Thread overview: 87+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-11-26 13:26 Memory again Carsten Mattner
2011-11-26 13:28 ` Carsten Mattner
2011-11-26 14:35 ` Dmitry Antipov
2011-11-26 14:48 ` Eli Zaretskii
2011-11-26 17:37 ` Dmitry Antipov
2011-11-26 20:19 ` Eli Zaretskii
2011-11-26 14:58 ` Carsten Mattner
2011-11-26 16:23 ` Eli Zaretskii
2011-11-26 19:02 ` Carsten Mattner
2011-11-26 20:31 ` Eli Zaretskii
2011-11-26 21:00 ` Eli Zaretskii
2011-11-27 10:29 ` Carsten Mattner
2011-11-27 10:43 ` Andreas Schwab
2011-11-27 13:53 ` Carsten Mattner
2011-11-27 13:11 ` Eli Zaretskii
2011-11-27 13:53 ` Carsten Mattner
2011-11-27 16:44 ` Eli Zaretskii
2011-11-27 17:37 ` Carsten Mattner
2011-11-27 17:59 ` Carsten Mattner
2011-12-06 4:02 ` Óscar Fuentes
2011-12-06 5:08 ` Eli Zaretskii
2011-12-06 9:35 ` Carsten Mattner
2011-12-06 10:24 ` Dmitry Antipov
2011-12-06 13:07 ` Eli Zaretskii
2011-12-06 13:29 ` Stefan Monnier
2011-12-06 17:20 ` Eli Zaretskii
2011-12-06 20:25 ` Stefan Monnier
2011-12-07 7:52 ` Eli Zaretskii
2011-12-07 8:15 ` Dmitry Antipov
2011-12-07 13:06 ` Eli Zaretskii
2011-12-07 14:01 ` Stefan Monnier
2011-12-08 17:30 ` Carsten Mattner
2011-12-09 3:39 ` Dmitry Antipov
2011-12-09 13:52 ` Carsten Mattner
2011-12-06 13:12 ` Eli Zaretskii
2011-12-06 16:28 ` Óscar Fuentes
2011-12-06 19:53 ` Stefan Monnier
2011-12-11 17:49 ` Nix
2011-12-15 3:52 ` Tim Connors
2011-12-15 4:09 ` Eli Zaretskii
2011-12-15 4:38 ` Tim Connors
2011-12-15 5:52 ` Eli Zaretskii
2011-12-15 4:50 ` Óscar Fuentes
2011-12-15 6:04 ` Eli Zaretskii
2011-12-16 21:55 ` Stefan Monnier
2011-12-17 17:40 ` Nix
2011-12-18 15:13 ` Dmitry Antipov
2011-12-19 1:34 ` Stefan Monnier [this message]
2011-12-19 8:28 ` Dmitry Antipov
2011-12-19 11:26 ` Stefan Monnier
2012-01-23 16:49 ` Nix
2012-01-25 16:19 ` Ted Zlatanov
2011-11-26 17:54 ` Dmitry Antipov
2011-11-26 18:47 ` martin rudalics
2011-11-26 19:09 ` Carsten Mattner
2011-11-28 4:27 ` Stefan Monnier
2011-11-28 9:24 ` Carsten Mattner
2011-11-28 15:31 ` Davis Herring
2011-11-28 21:33 ` Carsten Mattner
-- strict thread matches above, loose matches on Subject: below --
2011-12-19 19:51 emacs user
2011-12-20 5:32 ` Dmitry Antipov
2012-01-06 14:28 ` Chong Yidong
2012-01-06 15:53 ` emacs user
2011-12-20 6:34 emacs user
2011-12-20 7:43 ` Eli Zaretskii
2011-12-20 12:05 ` emacs user
2011-12-20 13:04 ` Eli Zaretskii
2011-12-20 22:07 ` Jan Djärv
2011-12-21 8:07 ` Jan Djärv
2011-12-21 10:39 ` Carsten Mattner
2011-12-21 17:55 ` emacs user
2011-12-22 14:08 ` Jan Djärv
2011-12-22 14:58 ` emacs user
2011-12-22 18:54 ` emacs user
2011-12-22 19:15 ` Jan Djärv
2011-12-23 4:41 ` YAMAMOTO Mitsuharu
2012-01-17 10:04 ` emacs user
2012-01-17 10:58 ` YAMAMOTO Mitsuharu
2012-01-17 13:14 ` emacs user
2012-01-18 1:30 ` YAMAMOTO Mitsuharu
2011-12-22 23:09 ` Carsten Mattner
2011-12-23 0:39 ` Stefan Monnier
2011-12-23 10:44 ` emacs user
2012-01-05 6:13 ` emacs user
2012-01-05 22:37 ` Jan Djärv
2012-01-06 9:58 ` emacs user
2012-01-06 11:10 ` Carsten Mattner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/emacs/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=jwvaa6p8i5i.fsf-monnier+emacs@gnu.org \
--to=monnier@iro.umontreal.ca \
--cc=dmantipov@yandex.ru \
--cc=emacs-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).