Re: Emacs bzr memory footprint

[Nix <nix@esperi.org.uk>]

On 21 Oct 2011, Dmitry Antipov uttered the following:
> Nevertheless, glibc malloc design assumes that smaller (below
> threshold above) chunks may be still allocated with mmap().

Really? I double-checked the glibc-2.12 malloc source, and mmap()s only
used for large allocations above that minimum threshold, for multiple
(threaded) arenas, and if sbrk() fails. (It normally uses a sliding
threshold to define 'minimum', but that threshold only goes *up* from
the minimum, never down. However, because Emacs sets the threshold
explicitly, the dynamic behaviour is halted and mmap() should be used
for all requests >64Kb.)

btw, regarding

,----[ src/alloc.c ]
| /* Specify maximum number of areas to mmap.  It would be nice to use a
|    value that explicitly means "no limit".  */
| 
| #define MMAP_MAX_AREAS 100000000
`----

there is a reason glibc malloc uses a default value of 65536, which is
simply that older Linux kernels used to get rather unhappy with programs
that defined many tens of thousands of VMAs (mmapped regions, roughly).
I think the first two programs to trigger this were Java and
User-Mode Linux: the latter mmap()ed one 4Kb chunk per 4Kb of guest
memory, and soon ended up causing exponential slowdowns inside the
kernel memory allocator.

More recent kernels (anything in the 2.6 line, certainly) don't have
this problem, so I think the huge value Emacs uses is probably fine. (If
the problem recurs, you *will* hear about it: it's hard to miss it when
a text editor gets stuck inside the kernel for several seconds inside
a not-too-large mmap(). But I suspect anyone still running a kernel old
enough to have this problem also has too little memory to see it with
regions of the size Emacs will mmap().)

> Run emacs with:
>
> strace -e signal=none -e trace=mmap,mremap,munmap emacs
>
> and you will see a lot of allocation requests of very different sizes.

Yep. However, a lot of those may come from other libraries: a good few
of them mmap() data in from files (especially at startup), mmap() small
anonymous regions and the like.

> On the other side, lisp data is newer mapped, and in some cases this
> is the main reason which causes sbrk()'ed heap fragmentation. Don't
> beat toolkits, don't beat system libraries. Now I'm observing the case
> where 4 GCs reduces amount of lisp data (live + free lists) from 30M
> to 8M, and mallinfo().fordblks raises from 1M to 22M. This 21M of
> fragmented heap is 'locked' into malloc freelists; they can be
> re-used, but will never released back to OS.

That's exactly what I feared :(

-- 
NULL && (void)