Re: Memory again

[Dmitry Antipov <dmantipov@yandex.ru>]

[-- Attachment #1: Type: text/plain, Size: 2320 bytes --]

On 12/19/2011 05:34 AM, Stefan Monnier wrote:

> `garbage-collect' is supposed to give that info.  At least it does give
> info about the data that's kept under alloc.c's control because
> of fragmentation (these are counted as "free cells").

I would like to propose a function which explicitly says how much free memory
the Emacs process holds. It's especially useful when there is a way to ask
system malloc about how much free memory is in the heap. This may be helpful
to distinguish real heap fragmentation from memory leaks and other misuses -
if the sum of values returned by 'memory-free' is, say, 10% of heap size,
then the real fragmentation enters into the game.

> Re-read what I wrote: I'm talking about the case where alloc.c free'd the
> data, but malloc did not munmap it.

I read it as 'even if we will fix allocation to use mmap and munmap, unmapped
memory would really be unused and still appear in process RSS, thus giving
higher RSS value than the process really has, until the memory pressure bump
it out of RAM'. That was obviously incorrect because RSS is recalculated
each time when mmap/mremap/munmap/brk is (successfully) called.

> It's easy to predict what it does in terms of "what is the benefit when
> I have a lot of fragmentation".  It's much more difficult to predict
> what it does in terms of "how is it going to affect Emacs's behavior for
> those 99% cases where it currently works just fine".

There is an interesting research paper about moving PLT Scheme from mark-sweep
GC to an accurate copying GC: http://www.cs.utah.edu/plt/publications/ismm09-rwrf.pdf.
In short, it's said about getting rid out of the most fragmentation issues at the cost
of 10-20% slowdown (although I consider their approach as practically inapplicable
beyond the pure research efforts).

> I agree that we're probably going to see better overall results by
> improving general memory use than by trying to attack
> fragmentation problems.

Among this list's subscribers, Nix <nix@[hidden]> is constantly reporting an enormous
memory usage caused by Gnus. Since I newer seen such a heap usage in my own test cases,
I offered him a 'cumulative' patch (against 24.0.92) with both immediate strings and
block vector allocator. He has said that we can expect the first results after Christmas :-).

Dmitry

[-- Attachment #2: memory_free.patch --]
[-- Type: text/plain, Size: 2175 bytes --]

=== modified file 'src/alloc.c'
--- src/alloc.c	2011-12-12 05:32:49 +0000
+++ src/alloc.c	2011-12-19 06:06:04 +0000
@@ -75,6 +75,10 @@
 
 #define MMAP_MAX_AREAS 100000000
 
+/* Value of mallinfo ().fordblks as seen at the end of last GC.  */
+
+static int bytes_free;
+
 #else /* not DOUG_LEA_MALLOC */
 
 /* The following come from gmalloc.c.  */
@@ -5271,6 +5275,10 @@
   total[7] = Fcons (make_number (total_strings),
 		    make_number (total_free_strings));
 
+#ifdef DOUG_LEA_MALLOC
+  bytes_free = mallinfo ().fordblks;
+#endif
+
 #if GC_MARK_STACK == GC_USE_GCPROS_CHECK_ZOMBIES
   {
     /* Compute average percentage of zombies.  */
@@ -6221,6 +6229,33 @@
   return end;
 }
 
+DEFUN ("memory-free", Fmemory_free, Smemory_free, 0, 0, 0,
+       doc: /* Return a list of two counters that measure how much free
+memory is hold by the Emacs process. Both counters are
+in KBytes. First counter shows how much memory holds in
+a free lists maintained by the Emacs itself. Second counter
+shows how much free memory is in the heap. If the second
+counter is zero, heap statistics is not available.  */)
+     (void)
+{
+  Lisp_Object data[2];
+  
+  data[0] = make_number
+    (min (MOST_POSITIVE_FIXNUM,
+	  (total_free_conses * sizeof (struct Lisp_Cons) +
+	   total_free_markers * sizeof (union Lisp_Misc) +
+	   total_free_symbols * sizeof (struct Lisp_Symbol) +
+	   total_free_floats * sizeof (struct Lisp_Float) +
+	   total_free_intervals * sizeof (struct interval) +
+	   total_free_strings * sizeof (struct Lisp_String)) / 1024));
+#ifdef DOUG_LEA_MALLOC
+  data[1] = make_number (min (MOST_POSITIVE_FIXNUM, bytes_free / 1024));
+#else
+  data[1] = make_number (0);
+#endif
+  return Flist (2, data);
+}
+
 DEFUN ("memory-use-counts", Fmemory_use_counts, Smemory_use_counts, 0, 0, 0,
        doc: /* Return a list of counters that measure how much consing there has been.
 Each of these counters increments for a certain kind of object.
@@ -6474,6 +6509,7 @@
   defsubr (&Spurecopy);
   defsubr (&Sgarbage_collect);
   defsubr (&Smemory_limit);
+  defsubr (&Smemory_free);
   defsubr (&Smemory_use_counts);
 
 #if GC_MARK_STACK == GC_USE_GCPROS_CHECK_ZOMBIES