Re: Emacs bzr memory footprint

[Dmitry Antipov <dmantipov@yandex.ru>]

On 10/21/2011 05:30 PM, Stefan Monnier wrote:

> Emacs already uses its own malloc for most objects: it uses malloc
> directly to allocate vectors (and this is actually something that we
> might want to change, because it (indirectly) comes with an O(log n)
> cost), but for strings, conses, markers, overlays, and floats it uses
> malloc only to get a chunk of memory which it then manages on its own.
> `memory-usage' does give you this kind of info when it says
> "5743392+864696 bytes in cons cells" which means 5MB of live cons-cells,
> and 800KB of cons-cells that are free (i.e. they are in a memory chunk
> that Emacs can't return to malloc because that chunk also contains live
> cons cells).

This creates two-level fragmentation: the whole heap is fragmented as seen
by malloc(), and allocated space is fragmented too, as seen by Emacs. Since
a lot of non-lisp allocation requests are satisfied from the same sbrk()ed
heap, here is the typical scenario: at startup, there is 10M of live lisp data.
Solid workload (e.g. visit 1000 files at once) increases this, say, to 100M.
When the workload is finished (kill all file-backed 1000 buffers), live lisp data
shrinks back to 10M. The rest 90M can't be freed because a) small percentage
is interleaved with live lisp data within the chunks managed by Emacs itself
and b) the rest, being freed by Emacs, can't be freed by malloc because
freed chunks are interleaved with non-freed, and all chunks are interleaved
with freed/lived lisp vectors and freed/lived non-lisp allocations. If next
solid workload will consume 100M of the heap, the most of this 90M will be re-used -
but it will never freed.

Due to all of the above, I suspect that the mostly-copying collector, where all
small (say, < 4K) lisp (and only lisp) allocation requests are satisfied from
1M-4M areas, might have substantial advantages over plain mark&sweep scheme.

Dmitry