From: Stefan Monnier <monnier@iro.umontreal.ca>
To: DJ Delorie <dj@redhat.com>
Cc: fweimer@redhat.com, carlos@redhat.com, hi-angel@yandex.ru,
45200@debbugs.gnu.org
Subject: bug#45200: [PATCH] Force Glibc to free the memory freed
Date: Wed, 03 Feb 2021 17:07:04 -0500 [thread overview]
Message-ID: <jwvwnvoet4e.fsf-monnier+emacs@gnu.org> (raw)
In-Reply-To: <xn7dnoevdo.fsf@greed.delorie.com> (DJ Delorie's message of "Wed, 03 Feb 2021 15:42:11 -0500")
>> I understand that as well. But I'm wondering why glibc is willing to
>> keep *indefinitely* an unused 200MB of memory, which is more than double
>> the mount of memory in use for the rest of the application's life.
> To be blunt, 200Mb is peanuts compared to some applications, and it's
I'm not worried about the absolute value, but about the proportion.
I think in memory management code, having an overall overhead of 50% is
generally considered acceptable (i.e. actual memory allocated is twice
the memory used by the application), whether that comes from
internal&external fragmentation or a stop© GC, ...
But in our specific use case, there seems to be no limit to the
overhead: if the application uses a heap of size N at some point in time
it will never grown back down, so the overhead can end up being
arbitrarily large.
> *nothing* compared to an enterprise application. Keeping 200M around to
> quickly satisfy memory requests of various sizes (not all cached chunks
> are the same size) is IMHO reasonable.
If the average allocation/deallocation rate justifies it, I fully agree.
But if the variation of allocated space stays well below that for a long
time, then those 200MB are truly wasted.
>> I mean I understand that you can't predict the future, but I expected
>> that "at some point" glibc should decide that those 200MB have been left
>> unused for long enough that they deserve to be returned to the OS.
> Where will we store that lifetime information?
I haven't thought very much about it, so I'm sure it's easy to shoot
holes through it, but I imagined something like:
- one `static unsigned long hoard_size` keeps the approximate amount of
space that is free but not returned to the OS.
Not sure where/when to keep it up to date cheaply, admittedly.
- one `static unsigned long smallest_recent_hoard_size`.
This is updated whenever we allocate memory from the OS.
- one `static unsigned long age_of_smallest_recent_hoard_size`.
This is incremented every time we allocate memory from the OS (and
reset whenever the value of smallest_recent_hoard_size is modified).
Then you'd call `malloc_trim` based on a magic formula combining
`age_of_smallest_recent_hoard_size` and the ratio of
`smallest_recent_hoard_size / total_heap_size` (and you'd trim only
what's necessary to release O(`smallest_recent_hoard_size`) memory).
> Yet another word of memory used,
Since 200MB is peanuts, I figure that extra 24B should be acceptable ;-)
> yet another syscall to check the time?
I didn't mean time as in an OS-notion of clock, no.
> I agree that we could do better at detecting long-unused chunks, but
> it's expensive (in terms of both development and runtime) to do so, and
> typically at the expense of some other desired metric.
No doubt.
> I would ask the Emacs devs why they wait until gc to free() memory
> instead of keeping track of uses more accurately and free()ing it
> right away. It's a similar type of compromise.
Delaying for some time is one thing. Delaying forever is another.
>> The doc of `malloc_trim` suggests it's occasionally called by `free` and
>> `mallopt` suggests via `M_TRIM_THRESHOLD` that there's a limit to how
>> much extra spare memory glibc keeps around, so this suggests that indeed
>> memory is trimmed "every once in a while".
> Only when the available memory is "at the top of the heap".
Ah, I see, that makes sense.
I do remember such behavior in other/older libc libraries.
> We used to have code that munmap()'d large "holes" in the cache,
That's what I seem to remember, indeed. And our memory management code
does play with `mallopt` in the hopes to encourage it to allocate using
`mmap` in the hopes that it then deallocates via `munmap`.
> but the resulting performance was horrible.
Hmm... so that explains why we're seeing those problems again.
Stefan
next prev parent reply other threads:[~2021-02-03 22:07 UTC|newest]
Thread overview: 88+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-12 18:43 bug#45200: Memory leaks: (garbage-collect) fails to reclaim memory Konstantin Kharlamov
2020-12-12 20:15 ` Eli Zaretskii
2020-12-12 22:44 ` Konstantin Kharlamov
2020-12-12 22:59 ` Lars Ingebrigtsen
2020-12-13 6:08 ` Eli Zaretskii
2020-12-13 5:53 ` Eli Zaretskii
2020-12-13 12:07 ` Konstantin Kharlamov
2021-01-24 15:24 ` bug#45200: [PATCH] Force Glibc to free the memory freed Konstantin Kharlamov
2021-01-24 15:40 ` Eli Zaretskii
2021-01-25 22:17 ` DJ Delorie
2021-01-25 22:28 ` Konstantin Kharlamov
2021-01-26 14:55 ` Eli Zaretskii
2021-01-26 15:02 ` Konstantin Kharlamov
2021-01-26 15:30 ` Stefan Monnier
2021-02-02 21:17 ` Konstantin Kharlamov
2021-02-03 4:45 ` Stefan Monnier
2021-02-03 4:50 ` Stefan Monnier
2021-02-03 6:04 ` Konstantin Kharlamov
2021-02-03 7:07 ` Eli Zaretskii
2021-02-03 7:15 ` Konstantin Kharlamov
2021-02-03 7:39 ` martin rudalics
2021-02-03 8:23 ` Konstantin Kharlamov
2021-02-03 9:35 ` martin rudalics
2021-02-03 9:49 ` Konstantin Kharlamov
2021-02-03 10:35 ` Konstantin Kharlamov
2021-02-03 11:06 ` martin rudalics
2021-02-03 11:08 ` Konstantin Kharlamov
2021-02-03 11:16 ` Konstantin Kharlamov
2021-02-03 12:56 ` martin rudalics
2021-02-03 13:00 ` Konstantin Kharlamov
2021-02-03 15:14 ` martin rudalics
2021-02-03 15:15 ` Stefan Monnier
2021-02-03 15:29 ` Konstantin Kharlamov
2021-02-03 16:02 ` Stefan Monnier
2021-02-03 16:35 ` Konstantin Kharlamov
2021-02-03 16:51 ` Stefan Monnier
2021-02-03 19:30 ` DJ Delorie
2021-02-03 19:36 ` DJ Delorie
2021-02-03 20:28 ` Konstantin Kharlamov
2021-02-03 20:51 ` DJ Delorie
2021-05-18 20:12 ` Konstantin Kharlamov
2021-05-19 4:11 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2021-05-19 4:26 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2021-05-19 6:46 ` Konstantin Kharlamov
2021-05-19 9:47 ` Eli Zaretskii
2021-05-19 9:55 ` Konstantin Kharlamov
2021-05-19 10:09 ` Eli Zaretskii
2021-02-03 14:51 ` Eli Zaretskii
2021-02-03 15:01 ` Konstantin Kharlamov
2021-02-03 14:44 ` Eli Zaretskii
2021-02-03 15:12 ` Andreas Schwab
2021-02-03 19:25 ` DJ Delorie
2021-02-03 19:49 ` Eli Zaretskii
2021-02-03 21:00 ` DJ Delorie
2021-02-03 20:24 ` Stefan Monnier
2021-02-03 20:42 ` DJ Delorie
2021-02-03 22:07 ` Stefan Monnier [this message]
2021-02-03 22:21 ` DJ Delorie
2021-02-03 23:32 ` Stefan Monnier
2021-02-04 0:31 ` DJ Delorie
2021-02-04 3:26 ` Stefan Monnier
2021-02-04 3:38 ` DJ Delorie
2021-02-04 3:55 ` Stefan Monnier
2021-02-04 4:02 ` DJ Delorie
2021-02-04 4:19 ` Stefan Monnier
2021-02-04 4:26 ` DJ Delorie
2021-02-04 4:04 ` DJ Delorie
2021-02-03 15:15 ` Stefan Monnier
2021-01-26 14:49 ` Eli Zaretskii
2021-01-26 16:13 ` DJ Delorie
2021-12-04 23:20 ` bug#45200: Memory leaks: (garbage-collect) fails to reclaim memory Lars Ingebrigtsen
2021-12-05 6:23 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2021-12-05 8:26 ` Eli Zaretskii
2022-05-01 9:43 ` bug#45200: Wishlist: There should be a `malloc-trim' function Lars Ingebrigtsen
2021-12-05 19:59 ` bug#45200: Memory leaks: (garbage-collect) fails to reclaim memory Lars Ingebrigtsen
2021-12-05 7:07 ` Eli Zaretskii
2021-01-24 18:51 ` Stefan Monnier
2021-01-24 19:00 ` Konstantin Kharlamov
2021-01-24 19:06 ` Konstantin Kharlamov
2021-01-24 19:55 ` Eli Zaretskii
2021-01-24 19:12 ` Stefan Monnier
2021-01-24 20:00 ` Konstantin Kharlamov
2021-01-24 20:11 ` Eli Zaretskii
2021-01-24 20:21 ` Konstantin Kharlamov
2021-01-24 21:20 ` Stefan Monnier
2021-01-24 21:26 ` Konstantin Kharlamov
2021-01-24 21:41 ` Stefan Monnier
2021-01-24 21:55 ` Konstantin Kharlamov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=jwvwnvoet4e.fsf-monnier+emacs@gnu.org \
--to=monnier@iro.umontreal.ca \
--cc=45200@debbugs.gnu.org \
--cc=carlos@redhat.com \
--cc=dj@redhat.com \
--cc=fweimer@redhat.com \
--cc=hi-angel@yandex.ru \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.