How to debug elisp memory leak

How to debug elisp memory leak

[Frode Vatvedt Fjeld]

I'm regularly getting the "lisp pointer size exceeded" warning after
about one week of running emacs. When this happens, nothing I do, such
as killing buffers, seems to help much; at most I can postpone the
inevitable crash for a few more minutes of work.

I figure there's a memory leak; somewhere there's e.g a global
variable with an ever-growing list of whatever. But I have no idea
where to start looking for what or where this might be. Can anyone
help me?

I'm using GNU Emacs 21.3.1, but this behavior has been consistent for
at least a few months and minor versions of emacs.

Thanks,
-- 
Frode Vatvedt Fjeld

Re: How to debug elisp memory leak

[Stefan Monnier]

> I'm regularly getting the "lisp pointer size exceeded" warning after
> about one week of running emacs. When this happens, nothing I do, such
> as killing buffers, seems to help much; at most I can postpone the
> inevitable crash for a few more minutes of work.

> I figure there's a memory leak; somewhere there's e.g a global
> variable with an ever-growing list of whatever. But I have no idea
> where to start looking for what or where this might be. Can anyone
> help me?

> I'm using GNU Emacs 21.3.1, but this behavior has been consistent for
> at least a few months and minor versions of Emacs.

I don't think there's a quick answer to this.
But for a start, can you try the pckage below?
It provides a command M-x memory-usage which will show what Emacs's heap is
made of (whic proportion is cons cells, buffer text, strings, arrays, ...).
It's not much info, but it's a start: when you get the "lisp pointer size
exceeded" thingy, hit M-x memory-usage and post the result here.
Also, try then to kill (lots of or even all) buffers and re-run M-x
memory-usage and post that result as well.


        Stefan

PS: Better to hit M-x memory-usage at least once before to make sure it
    works and is already loaded and ready to be used.


;;; memory-usage.el --- Analyze the memory usage of Emacs in various ways

;; Copyright (C) 2002, 2004  Free Software Foundation, Inc.

;; Author: Stefan Monnier <monnier@cs.yale.edu>
;; Keywords: maint

;; This file is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 2, or (at your option)
;; any later version.

;; This file is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to
;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
;; Boston, MA 02111-1307, USA.

;;; Commentary:

;; 

;;; Code:

(defun buffer-size-bytes (b)
  "Return total number of bytes in the buffer contents."
  (with-current-buffer b
    (save-restriction
      (widen)
      (- (position-bytes (point-max)) (position-bytes (point-min))))))

(defun buffer-gap-bytes (b)
  "Return total number of bytes in the buffer gap."
  (with-current-buffer b
    (gap-size)))

(defun buffer-total-bytes (b)
  "Return total number of ralloc bytes used by buffer."
  (with-current-buffer b
    (save-restriction
      (widen) 
      (+ (position-bytes (point-max))
         (- (position-bytes (point-min)))
         (gap-size)))))

;;;###autoload
(defun memory-usage ()
  "List all buffers and their memory usage."
  (interactive)
  (pop-to-buffer (get-buffer-create "*Buffer Details*"))
  (erase-buffer)
  (let* ((bufs (buffer-list))
         (num (length bufs))
         (gc-stats (garbage-collect))
         (conses (nth 0 gc-stats))
         (symbols (nth 1 gc-stats))
         (markers (nth 2 gc-stats))
         (strings (nth 3 gc-stats))
         (vectors (nth 4 gc-stats))
         (floats (nth 5 gc-stats))
         (intervals (nth 6 gc-stats)))
    (insert (format "Garbage collection stats:\n%s\n\n =>" gc-stats))
    (insert (format "\t%d bytes in cons cells\n" (* 8 (+ (car conses) (cdr conses)))))
    (insert (format "\t%d bytes in symbols\n" (* 24 (+ (car symbols) (cdr symbols)))))
    (insert (format "\t%d bytes in markers\n" (* 20 (+ (car markers) (cdr markers)))))
    (insert (format "\t%d bytes of string chars\n" strings))
    (insert (format "\t%d bytes of vector slots\n" (* 4 vectors)))
    (insert (format "\t%d bytes in floats\n" (* 12 (+ (car floats) (cdr floats)))))
    (insert (format "\t%d bytes in intervals\n" (* 28 (+ (car intervals) (cdr intervals)))))

    (insert (format "\nTotal bytes in lisp objects (not counting string and vector headers): %d\n\n"
                    (+ (* 8 (+ (car conses) (cdr conses)))
                       (* 24 (+ (car symbols) (cdr symbols)))
                       (* 20 (+ (car markers) (cdr markers)))
                       strings
                       vectors
                       (* 12 (+ (car floats) (cdr floats)))
                       (* 28 (+ (car intervals) (cdr intervals))))))

    (insert (format "Buffer ralloc memory usage:\n%d buffers\n%d bytes total (%d in gaps)\n"
                    num
                    (apply #'+ (mapcar #'buffer-total-bytes bufs))
                    (apply #'+ (mapcar #'buffer-gap-bytes bufs))))
    (insert (format "%10s\t%s\t%s\n\n" "Size" "Gap" "Name"))
    (insert (mapconcat
             (lambda (b)
               (format "%10d\t%s\t%s"
                       (buffer-size-bytes b)
                       (buffer-gap-bytes b)
                       (buffer-name b)))
             (sort bufs (lambda (b1 b2)
                          (> (buffer-size-bytes b1) (buffer-size-bytes b2))))
             "\n"))
    (insert "\n"))
  (goto-char (point-min)))


(provide 'memory-usage)
;; arch-tag: 04e012f0-3c59-4319-8d1a-e86204671ec5
;;; memory-usage.el ends here

Re: How to debug elisp memory leak

[Frode Vatvedt Fjeld]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I don't think there's a quick answer to this.  But for a start, can
> you try the pckage below?

I will, thank you.

-- 
Frode Vatvedt Fjeld

Re: How to debug elisp memory leak

[Frode Vatvedt Fjeld]

> Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I don't think there's a quick answer to this.  But for a start, can
>> you try the pckage below?

Frode Vatvedt Fjeld <frodef@cs.uit.no> writes:

> I will, thank you.

So, now the problem occurred, and I've got memory-usage loaded.

The output from a relatively fresh emacs (with most of my normal
work-set loaded) looked like this:

-------snip-------

Garbage collection stats:
((614832 . 32572) (42162 . 117) (2012 . 4693) 2899677 753431 (225 . 175) (76662 
. 771) (77466 . 5064))

 =>     5179232 bytes in cons cells
        1014696 bytes in symbols
        134100 bytes in markers
        2899677 bytes of string chars
        3013724 bytes of vector slots
        4800 bytes in floats
        2168124 bytes in intervals

Total bytes in lisp objects (not counting string and vector headers): 12154060

Buffer ralloc memory usage:
79 buffers
5012158 bytes total (3527923 in gaps)
      Size      Gap     Name

    253765      2000    compiler.lisp
    119948      2000    TAGS
    [...]
-------snip-------

Now, in emacs after I got the first "lisp pointer size exceeded"
warning, it looks like this:

-------snip-------
Garbage collection stats:
((803738 . 124208) (50225 . 55) (5454 . 5139) 2517494 981855 (238 . 343) (85714 . 2690) (98613 . 4896))

 =>	7423568 bytes in cons cells
	1206720 bytes in symbols
	211860 bytes in markers
	2517494 bytes of string chars
	3927420 bytes of vector slots
	6972 bytes in floats
	2475312 bytes in intervals

Total bytes in lisp objects (not counting string and vector headers): 14823781

Buffer ralloc memory usage:
126 buffers
10093087 bytes total (6913072 in gaps)
      Size	Gap	Name

   1272492	194988	 *Gnus Backlog*
    253765	2000	compiler.lisp
    119948	2000	TAGS
    [...]
-------snip-------

It seems to me that the "total bytes in lisp objects" has developed
quite reasonably, and the problem seems to be in the "ralloc"
stuff. In particular the *Gnus Backlog" thing. I ran "M-x
gnus-backlog-shutdown" which removed *Gnus Backlog*, and the
ralloc part of memory-usage became

-------snip-------
[...]
Total bytes in lisp objects (not counting string and vector headers): 14773166

Buffer ralloc memory usage:
124 buffers
8616381 bytes total (6721395 in gaps)
[...]
-------snip-------


Other than this, I can see that there are a number of buffers that are
heavily (re-)used by Gnus that are small in size, but with huge
gaps. For example, there's a buffer " *nntpd*" whose size is zero, but
whose gap is 1961195. 

Can I conclude that the problem has to do with these gaps? Is there an
easy way to tell emacs to re-compute gap-buffers or somesuch?

Thanks,
-- 
Frode Vatvedt Fjeld

Re: How to debug elisp memory leak

[Frode Vatvedt Fjeld]

Frode Vatvedt Fjeld <frodef@cs.uit.no> writes:

> Can I conclude that the problem has to do with these gaps? Is there
> an easy way to tell emacs to re-compute gap-buffers or somesuch?

..however, the "lisp pointer size" warning persists in popping up even
after I've killed all the buffers with big gaps, and the "ralloc"
memory reported by memory-usage is reduced to less than that of the
fresh emacs. Sigh.

-- 
Frode Vatvedt Fjeld

Re: How to debug elisp memory leak

[Frode Vatvedt Fjeld]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I don't think there's a quick answer to this.  But for a start, can
>> you try the pckage below?

I just googled around and found a message from you (Stefan Monnier) of
November 2003 that mentioned "a bad interaction with FreeBSD" as the
cause of what seems to be the same problem I'm seeing. I am indeed
using FreeBSD, and perhaps this was the crucial piece of information I
left out from my initial report.

The message I'm referring to is at
<URL:http://lists.gnu.org/archive/html/help-gnu-emacs/2003-11/msg00479.html>

You mentioned a fix in the works for this problem, what is the status
of this? My emacs is from /usr/ports as of june 1. 2004, and the
FreeBSD is 4.8-RELEASE.

Thanks,
-- 
Frode Vatvedt Fjeld

Re: How to debug elisp memory leak

[Florian Weimer]

* Frode Vatvedt Fjeld:

> I'm using GNU Emacs 21.3.1, but this behavior has been consistent for
> at least a few months and minor versions of emacs.

Recently, a memory leak in decode-coding-region (and other code that
uses some internal routines of coding.c) has been discovered.  It is
fixed in CVS (by Kenichi Handa).  I'm using the patch below for
version 21.3.

diff -urNad /home/fw/debian/tmp/emacs21-21.3+1/src/callproc.c emacs21-21.3+1/src/callproc.c
--- /home/fw/debian/tmp/emacs21-21.3+1/src/callproc.c	2002-07-09 02:02:36.000000000 +0200
+++ emacs21-21.3+1/src/callproc.c	2004-09-30 09:44:42.000000000 +0200
@@ -790,6 +790,8 @@
 		  {
 		    detect_coding (&process_coding, bufptr, nread);
 		    if (process_coding.composing != COMPOSITION_DISABLED)
+		      /* We have not yet allocated the composition
+			 data because the coding type was undecided.  */
 		      coding_allocate_composition_data (&process_coding, PT);
 		  }
 		if (process_coding.cmp_data)
diff -urNad /home/fw/debian/tmp/emacs21-21.3+1/src/coding.c emacs21-21.3+1/src/coding.c
--- /home/fw/debian/tmp/emacs21-21.3+1/src/coding.c	2003-03-16 23:06:55.000000000 +0100
+++ emacs21-21.3+1/src/coding.c	2004-09-30 09:44:42.000000000 +0200
@@ -5489,8 +5489,11 @@
 	coding_allocate_composition_data (coding, from);
     }
 
-  /* Try to skip the heading and tailing ASCIIs.  */
-  if (coding->type != coding_type_ccl)
+  /* Try to skip the heading and tailing ASCIIs.  We can't skip them
+     if we must run CCL program or there are compositions to
+     encode.  */
+  if (coding->type != coding_type_ccl
+      && (! coding->cmp_data || coding->cmp_data->used == 0))
     {
       int from_byte_orig = from_byte, to_byte_orig = to_byte;
 
@@ -5506,6 +5509,7 @@
 	  if (!replace)
 	    /* We must record and adjust for this new text now.  */
 	    adjust_after_insert (from, from_byte_orig, to, to_byte_orig, len);
+	  coding_free_composition_data (coding);
 	  return 0;
 	}
 
@@ -6106,12 +6110,16 @@
     coding_save_composition (coding, from, to, str);
 
   /* Try to skip the heading and tailing ASCIIs.  */
-  if (coding->type != coding_type_ccl)
+  if (coding->type != coding_type_ccl
+      && (! coding->cmp_data || coding->cmp_data->used == 0))
     {
       SHRINK_CONVERSION_REGION (&from, &to_byte, coding, XSTRING (str)->data,
 				1);
       if (from == to_byte)
-	return (nocopy ? str : Fcopy_sequence (str));
+ 	{
+ 	  coding_free_composition_data (coding);
+ 	  return (nocopy ? str : Fcopy_sequence (str));
+ 	}
       shrinked_bytes = from + (STRING_BYTES (XSTRING (str)) - to_byte);
     }
 
diff -urNad /home/fw/debian/tmp/emacs21-21.3+1/src/fileio.c emacs21-21.3+1/src/fileio.c
--- /home/fw/debian/tmp/emacs21-21.3+1/src/fileio.c	2003-02-04 11:52:40.000000000 +0100
+++ emacs21-21.3+1/src/fileio.c	2004-09-30 09:44:42.000000000 +0200
@@ -4087,7 +4087,7 @@
       if (how_much < 0)
 	{
 	  xfree (conversion_buffer);
-
+	  coding_free_composition_data (&coding);
 	  if (how_much == -1)
 	    error ("IO error reading %s: %s",
 		   XSTRING (orig_filename)->data, emacs_strerror (errno));
@@ -4109,6 +4109,7 @@
       if (bufpos == inserted)
 	{
 	  xfree (conversion_buffer);
+	  coding_free_composition_data (&coding);
 	  emacs_close (fd);
 	  specpdl_ptr--;
 	  /* Truncate the buffer to the size of the file.  */
diff -urNad /home/fw/debian/tmp/emacs21-21.3+1/src/process.c emacs21-21.3+1/src/process.c
--- /home/fw/debian/tmp/emacs21-21.3+1/src/process.c	2003-03-16 23:06:56.000000000 +0100
+++ emacs21-21.3+1/src/process.c	2004-09-30 09:44:42.000000000 +0200
@@ -3347,6 +3347,7 @@
       object = XPROCESS (proc)->encoding_buf;
       encode_coding (coding, (char *) buf, XSTRING (object)->data,
 		     len, STRING_BYTES (XSTRING (object)));
+      coding_free_composition_data (coding);
       len = coding->produced;
       buf = XSTRING (object)->data;
       if (temp_buf)

Re: How to debug elisp memory leak

[Stefan Monnier]

> You mentioned a fix in the works for this problem, what is the status
> of this? My emacs is from /usr/ports as of june 1. 2004, and the
> FreeBSD is 4.8-RELEASE.

The fix for that problem is installed in the Emacs-CVS trunk (and will thus
be in the next major Emacs release).

Of course, along with the memory-usage.el, it's always worthwhile to check
the size(s) of the process as reported by `ps' (i.e. things like VM and RSS).

7MB of gaps for 10MB of memory is clearly not an efficient use of memory
(and IIRC the Emacs-CVS code should perform better in this respect as well),
but 10MB should not be a cause for a "lisp pointer size" warning (which
should be closer to the 100MB order of magnitude on your Emacs).  So maybe
it's got to do with FreeBSD's memory layout which interacts badly with
Emacs's, or it can be that the extra memory used is not found by
memory-usage.el (which is why the `ps' stats are important).


        Stefan