* Debugging memory leaks/stale references @ 2004-09-21 17:38 Florian Weimer 2004-09-21 19:49 ` Simon Josefsson 2004-09-21 19:57 ` Stefan Monnier 0 siblings, 2 replies; 15+ messages in thread From: Florian Weimer @ 2004-09-21 17:38 UTC (permalink / raw) I've decided that it's time to track down an annoying bug in Gnus (or Emacs). The symptom is that memory-limit keeps growing and growing, but the data segment size (as allocated from the operating system) remains approximately constant. In my case, this behavior is exposed mainly when entering large NNTP groups with Gnus. Is it possible to dump all reachable Lisp objects to a file, in the format that is returned by princ? I could run diff(1) on two versions of the file, and this might reveal which objects are leaking (or held by stale references). garbage-collect output alone is not very helpful. (I'm going to test CVS Emacs soon, just to be sure that the behavior is still the same. It's probably a Gnus bug, but you never know.) ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-21 17:38 Debugging memory leaks/stale references Florian Weimer @ 2004-09-21 19:49 ` Simon Josefsson 2004-09-27 19:40 ` Florian Weimer 2004-09-21 19:57 ` Stefan Monnier 1 sibling, 1 reply; 15+ messages in thread From: Simon Josefsson @ 2004-09-21 19:49 UTC (permalink / raw) Florian Weimer <fw@deneb.enyo.de> writes: > I've decided that it's time to track down an annoying bug in Gnus (or > Emacs). The symptom is that memory-limit keeps growing and growing, > but the data segment size (as allocated from the operating system) > remains approximately constant. In my case, this behavior is exposed > mainly when entering large NNTP groups with Gnus. > > Is it possible to dump all reachable Lisp objects to a file, in the > format that is returned by princ? I could run diff(1) on two versions > of the file, and this might reveal which objects are leaking (or held > by stale references). garbage-collect output alone is not very > helpful. > > (I'm going to test CVS Emacs soon, just to be sure that the behavior > is still the same. It's probably a Gnus bug, but you never know.) I'm using CVS Emacs, and can confirm that the behavior is the same. When I notice that emacs feel sluggish, it is often the case that ps report that emacs is using nearly all physical amount memory (1GB here). Tracking down this would be good. I think someone said that disabling the gnus agent at least reduce the problem. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-21 19:49 ` Simon Josefsson @ 2004-09-27 19:40 ` Florian Weimer 2004-09-27 19:52 ` Stefan Monnier 2004-09-28 15:20 ` Richard Stallman 0 siblings, 2 replies; 15+ messages in thread From: Florian Weimer @ 2004-09-27 19:40 UTC (permalink / raw) * Simon Josefsson: > I'm using CVS Emacs, and can confirm that the behavior is the same. > When I notice that emacs feel sluggish, it is often the case that ps > report that emacs is using nearly all physical amount memory (1GB > here). Tracking down this would be good. I think someone said that > disabling the gnus agent at least reduce the problem. Okay, I've done some debugging, mainly by instrumenting the garbage collector. Basically, the patch below is a hook into mark_object() and prints object types, address, and also contents (for symbols) while they are traversed. Unfortunately, no major leak turned up, just a few thousand cons cells which can't cause the massive leak I see (in the order of dozens of megabytes after entering/exiting a large NNTP group with Gnus). So I'm back to the drawing board and have a few further questions. Is it possible to run a full-featured Emacs (including X11 support) on a target that does not support dumping? (In case you wonder, x86/valgrind is such a target. 8-) Is there a method to determine the (approximate) size of a buffer? Are there any other objects that can change their size after allocation? (I'm pretty sure that there are no additional Lisp objects allocated, but maybe an existing object grows without bounds.) Oh, and for your amusement, I've appended by debugging patch below. Basically, this is straight from the "you do not want to need this, really" department. Of course, it's not intended for inclusion in to Emacs (except for the first hunk, maybe). --- orig/src/.arch-inventory +++ mod/src/.arch-inventory @@ -4,6 +4,6 @@ # Auto-generated files, which ignore precious ^(config\.stamp|config\.h|epaths\.h)$ -backup ^(stamp-oldxmenu|prefix-args|temacs|emacs|emacs-[0-9.]*)$ +backup ^(stamp-oldxmenu|prefix-args|temacs|emacs|TAGS-LISP|emacs-[0-9.]*)$ # arch-tag: 277cc7ae-b3f5-44af-abf1-84c073164543 --- orig/src/alloc.c +++ mod/src/alloc.c @@ -258,12 +258,15 @@ Lisp_Object Vgc_elapsed; /* accumulated elapsed time in GC */ EMACS_INT gcs_done; /* accumulated GCs */ -static void mark_buffer P_ ((Lisp_Object)); -extern void mark_kboards P_ ((void)); -extern void mark_backtrace P_ ((void)); +/* If non-zero, dump objects to stderr while they are marked. */ +static int do_dump = 0; + +static void dump_marker_section (char *name); + +static void mark_buffer P_ ((Lisp_Object, unsigned)); static void gc_sweep P_ ((void)); -static void mark_glyph_matrix P_ ((struct glyph_matrix *)); -static void mark_face_cache P_ ((struct face_cache *)); +static void mark_glyph_matrix P_ ((struct glyph_matrix *, unsigned)); +static void mark_face_cache P_ ((struct face_cache *, unsigned)); #ifdef HAVE_WINDOW_SYSTEM static void mark_image P_ ((struct image *)); @@ -275,8 +278,6 @@ static void free_large_strings P_ ((void)); static void sweep_strings P_ ((void)); -extern int message_enable_multibyte; - /* When scanning the C stack for live Lisp objects, Emacs keeps track of what memory allocated via lisp_malloc is intended for what purpose. This enumeration specifies the type of memory. */ @@ -389,8 +390,8 @@ static int live_symbol_p P_ ((struct mem_node *, void *)); static int live_float_p P_ ((struct mem_node *, void *)); static int live_misc_p P_ ((struct mem_node *, void *)); -static void mark_maybe_object P_ ((Lisp_Object)); -static void mark_memory P_ ((void *, void *)); +static void mark_maybe_object P_ ((Lisp_Object, unsigned)); +static void mark_memory P_ ((void *, void *, unsigned)); static void mem_init P_ ((void)); static struct mem_node *mem_insert P_ ((void *, void *, enum mem_type)); static void mem_insert_fixup P_ ((struct mem_node *)); @@ -1217,7 +1218,7 @@ { eassert (!i->gcmarkbit); /* Intervals are never shared. */ i->gcmarkbit = 1; - mark_object (i->plist); + mark_object (i->plist, 0); } @@ -3688,8 +3689,9 @@ /* Mark OBJ if we can prove it's a Lisp_Object. */ static INLINE void -mark_maybe_object (obj) +mark_maybe_object (obj, depth) Lisp_Object obj; + unsigned depth; { void *po = (void *) XPNTR (obj); struct mem_node *m = mem_find (po); @@ -3743,7 +3745,7 @@ zombies[nzombies] = obj; ++nzombies; #endif - mark_object (obj); + mark_object (obj, depth + 1); } } } @@ -3753,8 +3755,9 @@ marked. */ static INLINE void -mark_maybe_pointer (p) +mark_maybe_pointer (p, depth) void *p; + unsigned depth; { struct mem_node *m; @@ -3824,7 +3827,7 @@ } if (!GC_NILP (obj)) - mark_object (obj); + mark_object (obj, depth + 1); } } @@ -3832,8 +3835,9 @@ /* Mark Lisp objects referenced from the address range START..END. */ static void -mark_memory (start, end) +mark_memory (start, end, depth) void *start, *end; + unsigned depth; { Lisp_Object *p; void **pp; @@ -3853,7 +3857,7 @@ /* Mark Lisp_Objects. */ for (p = (Lisp_Object *) start; (void *) p < end; ++p) - mark_maybe_object (*p); + mark_maybe_object (*p, depth + 1); /* Mark Lisp data pointed to. This is necessary because, in some situations, the C compiler optimizes Lisp objects away, so that @@ -4098,7 +4102,7 @@ #endif #endif for (i = 0; i < sizeof (Lisp_Object); i += GC_LISP_OBJECT_ALIGNMENT) - mark_memory ((char *) stack_base + i, end); + mark_memory ((char *) stack_base + i, end, 1); #if GC_MARK_STACK == GC_MARK_STACK_CHECK_GCPROS check_gcpros (); @@ -4455,14 +4459,14 @@ /* Mark all the special slots that serve as the roots of accessibility. */ for (i = 0; i < staticidx; i++) - mark_object (*staticvec[i]); + mark_object (*staticvec[i], 1); for (bind = specpdl; bind != specpdl_ptr; bind++) { - mark_object (bind->symbol); - mark_object (bind->old_value); + mark_object (bind->symbol, 1); + mark_object (bind->old_value, 1); } - mark_kboards (); + mark_kboards (1); #ifdef USE_GTK { @@ -4483,18 +4487,22 @@ } #endif + dump_marker_section ("Begin marking byte stack."); mark_byte_stack (); + dump_marker_section ("Begin marking byte stack."); for (catch = catchlist; catch; catch = catch->next) { - mark_object (catch->tag); - mark_object (catch->val); + mark_object (catch->tag, 1); + mark_object (catch->val, 1); } for (handler = handlerlist; handler; handler = handler->next) { - mark_object (handler->handler); - mark_object (handler->var); + mark_object (handler->handler, 1); + mark_object (handler->var, 1); } + dump_marker_section ("Begin marking backtrace."); mark_backtrace (); + dump_marker_section ("End marking backtrace."); #if GC_MARK_STACK == GC_USE_GCPROS_CHECK_ZOMBIES mark_stack (); @@ -4542,7 +4550,7 @@ } /* Now that we have stripped the elements that need not be in the undo_list any more, we can finally mark the list. */ - mark_object (nextb->undo_list); + mark_object (nextb->undo_list, 1); nextb = nextb->next; } @@ -4630,13 +4638,24 @@ return Flist (sizeof total / sizeof *total, total); } +DEFUN ("garbage-collect-dump", Fgarbage_collect_dump, Sgarbage_collect_dump, 0, 0, "", + doc: /* Run garbage collection and dump objects. */) + () +{ + ++do_dump; + dump_marker_section ("Begin garbage collection."); + Fgarbage_collect (); + dump_marker_section ("End garbage collection."); + --do_dump; +} /* Mark Lisp objects in glyph matrix MATRIX. Currently the only interesting objects referenced from glyphs are strings. */ static void -mark_glyph_matrix (matrix) +mark_glyph_matrix (matrix, depth) struct glyph_matrix *matrix; + unsigned depth; { struct glyph_row *row = matrix->rows; struct glyph_row *end = row + matrix->nrows; @@ -4653,7 +4672,7 @@ for (; glyph < end_glyph; ++glyph) if (GC_STRINGP (glyph->object) && !STRING_MARKED_P (XSTRING (glyph->object))) - mark_object (glyph->object); + mark_object (glyph->object, depth + 1); } } } @@ -4662,8 +4681,9 @@ /* Mark Lisp faces in the face cache C. */ static void -mark_face_cache (c) +mark_face_cache (c, depth) struct face_cache *c; + unsigned depth; { if (c) { @@ -4675,7 +4695,7 @@ if (face) { for (j = 0; j < LFACE_VECTOR_SIZE; ++j) - mark_object (face->lface[j]); + mark_object (face->lface[j], depth + 1); } } } @@ -4690,10 +4710,10 @@ mark_image (img) struct image *img; { - mark_object (img->spec); + mark_object (img->spec, 1); if (!NILP (img->data.lisp_val)) - mark_object (img->data.lisp_val); + mark_object (img->data.lisp_val, 1); } @@ -4725,9 +4745,84 @@ Normally this is zero and the check never goes off. */ int mark_object_loop_halt; +/* For memory debugging: dumps the string in human-readable form to + stderr. */ +static void +dump_string (char *data, unsigned length) +{ + unsigned i; + + for (i = 0; i < length; ++i) + { + char c = data[i]; + if (c >= 32 && c <= 126) + fputc (c, stderr); + else if (c == '"' || c == '\\') + { + fputc ('\\', stderr); + fputc (c, stderr); + } + else if (c == '\n') + fputs ("\\n", stderr); + else if (c == '\t') + fputs ("\\t", stderr); + else if (c == '\r') + fputs ("\\r", stderr); + else + fprintf (stderr, "\\%03o", (unsigned)(unsigned char) c); + } +} + +/* For memory debugging: prints OBJ of TYPE, at nesting level + DEPTH to stderr. */ +static void +dump_object (char *type, unsigned depth, Lisp_Object obj, int marked) +{ + unsigned i; + + if (!do_dump) + return; + + fprintf (stderr, "[%u] %s (%p)%s\n", + depth, type, (void *)obj, marked ? " *" : ""); +} + +static void +dump_object_int (unsigned depth, Lisp_Object obj) +{ + if (!do_dump) + return; + + fprintf (stderr, "[%u] INT %d\n", depth, XINT(obj)); +} + +static void +dump_marker_section(char *name) +{ + if (!do_dump) + return; + + fprintf (stderr, "%s\n", name); +} + +/* For memory debugging: prints OBJ of TYPE, with NAME, at nesting + level DEPTH to stderr. */ +static void +dump_object_name (char *type, unsigned depth, Lisp_Object obj, struct Lisp_String* name, int marked) +{ + if (!do_dump) + return; + + fprintf (stderr, "[%u] %s (%p) \"", depth, type, (void *)obj); + dump_string (name->data, name->size & ~ARRAY_MARK_FLAG); + fprintf (stderr, "\"%s\n", marked ? " *" : ""); +} + + void -mark_object (arg) +mark_object (arg, depth) Lisp_Object arg; + unsigned depth; { register Lisp_Object obj = arg; #ifdef GC_CHECK_MARKED_OBJECTS @@ -4792,6 +4887,7 @@ CHECK_ALLOCATED_AND_LIVE (live_string_p); MARK_INTERVAL_TREE (ptr->intervals); MARK_STRING (ptr); + dump_object_name ("STRING", depth, obj, ptr, ptr->size & ARRAY_MARK_FLAG); #ifdef GC_CHECK_STRING_BYTES /* Check that the string size recorded in the string is the same as the one recorded in the sdata structure. */ @@ -4811,6 +4907,8 @@ if (GC_BUFFERP (obj)) { + dump_object ("BUFFER", depth, obj, VECTOR_MARKED_P (XBUFFER (obj))); + if (!VECTOR_MARKED_P (XBUFFER (obj))) { #ifdef GC_CHECK_MARKED_OBJECTS @@ -4823,7 +4921,7 @@ abort (); } #endif /* GC_CHECK_MARKED_OBJECTS */ - mark_buffer (obj); + mark_buffer (obj, depth + 1); } } else if (GC_SUBRP (obj)) @@ -4837,6 +4935,8 @@ register EMACS_INT size = ptr->size; register int i; + dump_object ("COMPILED", depth, obj, VECTOR_MARKED_P (ptr)); + if (VECTOR_MARKED_P (ptr)) break; /* Already marked */ @@ -4846,7 +4946,7 @@ for (i = 0; i < size; i++) /* and then mark its elements */ { if (i != COMPILED_CONSTANTS) - mark_object (ptr->contents[i]); + mark_object (ptr->contents[i], depth + 1); } obj = ptr->contents[COMPILED_CONSTANTS]; goto loop; @@ -4855,40 +4955,48 @@ { register struct frame *ptr = XFRAME (obj); + dump_object ("FRAME", depth, obj, VECTOR_MARKED_P (ptr)); + if (VECTOR_MARKED_P (ptr)) break; /* Already marked */ + VECTOR_MARK (ptr); /* Else mark it */ CHECK_LIVE (live_vector_p); - mark_object (ptr->name); - mark_object (ptr->icon_name); - mark_object (ptr->title); - mark_object (ptr->focus_frame); - mark_object (ptr->selected_window); - mark_object (ptr->minibuffer_window); - mark_object (ptr->param_alist); - mark_object (ptr->scroll_bars); - mark_object (ptr->condemned_scroll_bars); - mark_object (ptr->menu_bar_items); - mark_object (ptr->face_alist); - mark_object (ptr->menu_bar_vector); - mark_object (ptr->buffer_predicate); - mark_object (ptr->buffer_list); - mark_object (ptr->menu_bar_window); - mark_object (ptr->tool_bar_window); - mark_face_cache (ptr->face_cache); + mark_object (ptr->name, depth + 1); + mark_object (ptr->icon_name, depth + 1); + mark_object (ptr->title, depth + 1); + mark_object (ptr->focus_frame, depth + 1); + mark_object (ptr->selected_window, depth + 1); + mark_object (ptr->minibuffer_window, depth + 1); + mark_object (ptr->param_alist, depth + 1); + mark_object (ptr->scroll_bars, depth + 1); + mark_object (ptr->condemned_scroll_bars, depth + 1); + mark_object (ptr->menu_bar_items, depth + 1); + mark_object (ptr->face_alist, depth + 1); + mark_object (ptr->menu_bar_vector, depth + 1); + mark_object (ptr->buffer_predicate, depth + 1); + mark_object (ptr->buffer_list, depth + 1); + mark_object (ptr->menu_bar_window, depth + 1); + mark_object (ptr->tool_bar_window, depth + 1); + mark_face_cache (ptr->face_cache, depth + 1); #ifdef HAVE_WINDOW_SYSTEM + dump_marker_section ("Begin marking FRAME images."); mark_image_cache (ptr); - mark_object (ptr->tool_bar_items); - mark_object (ptr->desired_tool_bar_string); - mark_object (ptr->current_tool_bar_string); + dump_marker_section ("End marking FRAME images."); + mark_object (ptr->tool_bar_items, depth + 1); + mark_object (ptr->desired_tool_bar_string, depth + 1); + mark_object (ptr->current_tool_bar_string, depth + 1); #endif /* HAVE_WINDOW_SYSTEM */ } else if (GC_BOOL_VECTOR_P (obj)) { register struct Lisp_Vector *ptr = XVECTOR (obj); + dump_object ("BOOL_VECTOR", depth, obj, VECTOR_MARKED_P (ptr)); + if (VECTOR_MARKED_P (ptr)) break; /* Already marked */ + CHECK_LIVE (live_vector_p); VECTOR_MARK (ptr); /* Else mark it */ } @@ -4898,6 +5006,8 @@ struct window *w = XWINDOW (obj); register int i; + dump_object ("WINDOW", depth, obj, VECTOR_MARKED_P (ptr)); + /* Stop if already marked. */ if (VECTOR_MARKED_P (ptr)) break; @@ -4911,7 +5021,7 @@ for (i = 0; (char *) &ptr->contents[i] < (char *) &w->current_matrix; i++) - mark_object (ptr->contents[i]); + mark_object (ptr->contents[i], depth + 1); /* Mark glyphs for leaf windows. Marking window matrices is sufficient because frame matrices use the same glyph @@ -4920,14 +5030,16 @@ && NILP (w->vchild) && w->current_matrix) { - mark_glyph_matrix (w->current_matrix); - mark_glyph_matrix (w->desired_matrix); + mark_glyph_matrix (w->current_matrix, depth + 1); + mark_glyph_matrix (w->desired_matrix, depth + 1); } } else if (GC_HASH_TABLE_P (obj)) { struct Lisp_Hash_Table *h = XHASH_TABLE (obj); + dump_object ("HASH_TABLE", depth, obj, VECTOR_MARKED_P (h)); + /* Stop if already marked. */ if (VECTOR_MARKED_P (h)) break; @@ -4941,20 +5053,20 @@ Being in the next_weak chain should not keep the hash table alive. No need to mark `count' since it is an integer. */ - mark_object (h->test); - mark_object (h->weak); - mark_object (h->rehash_size); - mark_object (h->rehash_threshold); - mark_object (h->hash); - mark_object (h->next); - mark_object (h->index); - mark_object (h->user_hash_function); - mark_object (h->user_cmp_function); + mark_object (h->test, depth + 1); + mark_object (h->weak, depth + 1); + mark_object (h->rehash_size, depth + 1); + mark_object (h->rehash_threshold, depth + 1); + mark_object (h->hash, depth + 1); + mark_object (h->next, depth + 1); + mark_object (h->index, depth + 1); + mark_object (h->user_hash_function, depth + 1); + mark_object (h->user_cmp_function, depth + 1); /* If hash table is not weak, mark all keys and values. For weak tables, mark only the vector. */ if (GC_NILP (h->weak)) - mark_object (h->key_and_value); + mark_object (h->key_and_value, depth + 1); else VECTOR_MARK (XVECTOR (h->key_and_value)); } @@ -4964,6 +5076,8 @@ register EMACS_INT size = ptr->size; register int i; + dump_object ("VECTOR", depth, obj, VECTOR_MARKED_P (ptr)); + if (VECTOR_MARKED_P (ptr)) break; /* Already marked */ CHECK_LIVE (live_vector_p); VECTOR_MARK (ptr); /* Else mark it */ @@ -4971,7 +5085,7 @@ size &= PSEUDOVECTOR_SIZE_MASK; for (i = 0; i < size; i++) /* and then mark its elements */ - mark_object (ptr->contents[i]); + mark_object (ptr->contents[i], depth + 1); } break; @@ -4980,12 +5094,15 @@ register struct Lisp_Symbol *ptr = XSYMBOL (obj); struct Lisp_Symbol *ptrx; + dump_object_name ("SYMBOL", depth, obj, XSTRING (ptr->xname), ptr->gcmarkbit); + if (ptr->gcmarkbit) break; + CHECK_ALLOCATED_AND_LIVE (live_symbol_p); ptr->gcmarkbit = 1; - mark_object (ptr->value); - mark_object (ptr->function); - mark_object (ptr->plist); + mark_object (ptr->value, depth + 1); + mark_object (ptr->function, depth + 1); + mark_object (ptr->plist, depth + 1); if (!PURE_POINTER_P (XSTRING (ptr->xname))) MARK_STRING (XSTRING (ptr->xname)); @@ -5006,6 +5123,7 @@ case Lisp_Misc: CHECK_ALLOCATED_AND_LIVE (live_misc_p); + dump_object ("MISC", depth, obj, XMARKER (obj)->gcmarkbit); if (XMARKER (obj)->gcmarkbit) break; XMARKER (obj)->gcmarkbit = 1; @@ -5023,9 +5141,9 @@ obj = ptr->realvalue; goto loop; } - mark_object (ptr->realvalue); - mark_object (ptr->buffer); - mark_object (ptr->frame); + mark_object (ptr->realvalue, depth + 1); + mark_object (ptr->buffer, depth + 1); + mark_object (ptr->frame, depth + 1); obj = ptr->cdr; goto loop; } @@ -5058,18 +5176,17 @@ Lisp_Object *p = (Lisp_Object *) ptr->pointer; int nelt; for (nelt = ptr->integer; nelt > 0; nelt--, p++) - mark_maybe_object (*p); + mark_maybe_object (*p, depth + 1); } } #endif break; - case Lisp_Misc_Overlay: { struct Lisp_Overlay *ptr = XOVERLAY (obj); - mark_object (ptr->start); - mark_object (ptr->end); - mark_object (ptr->plist); + mark_object (ptr->start, depth + 1); + mark_object (ptr->end, depth + 1); + mark_object (ptr->plist, depth + 1); if (ptr->next) { XSETMISC (obj, ptr->next); @@ -5086,30 +5203,23 @@ case Lisp_Cons: { register struct Lisp_Cons *ptr = XCONS (obj); + dump_object ("CONS", depth, obj, CONS_MARKED_P (ptr)); if (CONS_MARKED_P (ptr)) break; CHECK_ALLOCATED_AND_LIVE (live_cons_p); CONS_MARK (ptr); - /* If the cdr is nil, avoid recursion for the car. */ - if (EQ (ptr->cdr, Qnil)) - { - obj = ptr->car; - cdr_count = 0; - goto loop; - } - mark_object (ptr->car); - obj = ptr->cdr; - cdr_count++; - if (cdr_count == mark_object_loop_halt) - abort (); - goto loop; + mark_object (ptr->car, depth + 1); + mark_object (ptr->cdr, depth + 1); + break; } case Lisp_Float: CHECK_ALLOCATED_AND_LIVE (live_float_p); + dump_object ("CONS", depth, obj, FLOAT_MARKED_P (XFLOAT (obj))); FLOAT_MARK (XFLOAT (obj)); break; case Lisp_Int: + dump_object_int (depth, obj); break; default: @@ -5124,8 +5234,9 @@ /* Mark the pointers in a buffer structure. */ static void -mark_buffer (buf) +mark_buffer (buf, depth) Lisp_Object buf; + unsigned depth; { register struct buffer *buffer = XBUFFER (buf); register Lisp_Object *ptr, tmp; @@ -5142,24 +5253,24 @@ if (buffer->overlays_before) { XSETMISC (tmp, buffer->overlays_before); - mark_object (tmp); + mark_object (tmp, depth + 1); } if (buffer->overlays_after) { XSETMISC (tmp, buffer->overlays_after); - mark_object (tmp); + mark_object (tmp, depth + 1); } for (ptr = &buffer->name; (char *)ptr < (char *)buffer + sizeof (struct buffer); ptr++) - mark_object (*ptr); + mark_object (*ptr, depth + 1); /* If this is an indirect buffer, mark its base buffer. */ if (buffer->base_buffer && !VECTOR_MARKED_P (buffer->base_buffer)) { XSETBUFFER (base_buffer, buffer->base_buffer); - mark_buffer (base_buffer); + mark_buffer (base_buffer, depth + 1); } } @@ -5792,6 +5903,7 @@ defsubr (&Smake_marker); defsubr (&Spurecopy); defsubr (&Sgarbage_collect); + defsubr (&Sgarbage_collect_dump); defsubr (&Smemory_limit); defsubr (&Smemory_use_counts); --- orig/src/bytecode.c +++ mod/src/bytecode.c @@ -289,10 +289,10 @@ eassert (stack->top); for (obj = stack->bottom; obj <= stack->top; ++obj) - mark_object (*obj); + mark_object (*obj, 1); - mark_object (stack->byte_string); - mark_object (stack->constants); + mark_object (stack->byte_string, 1); + mark_object (stack->constants, 1); } } --- orig/src/eval.c +++ mod/src/eval.c @@ -3260,14 +3260,14 @@ for (backlist = backtrace_list; backlist; backlist = backlist->next) { - mark_object (*backlist->function); + mark_object (*backlist->function, 1); if (backlist->nargs == UNEVALLED || backlist->nargs == MANY) i = 0; else i = backlist->nargs - 1; for (; i >= 0; i--) - mark_object (backlist->args[i]); + mark_object (backlist->args[i], 1); } } --- orig/src/fns.c +++ mod/src/fns.c @@ -4804,13 +4804,13 @@ /* Make sure key and value survive. */ if (!key_known_to_survive_p) { - mark_object (HASH_KEY (h, i)); + mark_object (HASH_KEY (h, i), 1); marked = 1; } if (!value_known_to_survive_p) { - mark_object (HASH_VALUE (h, i)); + mark_object (HASH_VALUE (h, i), 1); marked = 1; } } --- orig/src/keyboard.c +++ mod/src/keyboard.c @@ -11425,7 +11425,8 @@ /* Mark the pointers in the kboard objects. Called by the Fgarbage_collector. */ void -mark_kboards () +mark_kboards (depth) + unsigned depth; { KBOARD *kb; Lisp_Object *p; @@ -11433,19 +11434,19 @@ { if (kb->kbd_macro_buffer) for (p = kb->kbd_macro_buffer; p < kb->kbd_macro_ptr; p++) - mark_object (*p); - mark_object (kb->Voverriding_terminal_local_map); - mark_object (kb->Vlast_command); - mark_object (kb->Vreal_last_command); - mark_object (kb->Vprefix_arg); - mark_object (kb->Vlast_prefix_arg); - mark_object (kb->kbd_queue); - mark_object (kb->defining_kbd_macro); - mark_object (kb->Vlast_kbd_macro); - mark_object (kb->Vsystem_key_alist); - mark_object (kb->system_key_syms); - mark_object (kb->Vdefault_minibuffer_frame); - mark_object (kb->echo_string); + mark_object (*p, depth + 1); + mark_object (kb->Voverriding_terminal_local_map, depth + 1); + mark_object (kb->Vlast_command, depth + 1); + mark_object (kb->Vreal_last_command, depth + 1); + mark_object (kb->Vprefix_arg, depth + 1); + mark_object (kb->Vlast_prefix_arg, depth + 1); + mark_object (kb->kbd_queue, depth + 1); + mark_object (kb->defining_kbd_macro, depth + 1); + mark_object (kb->Vlast_kbd_macro, depth + 1); + mark_object (kb->Vsystem_key_alist, depth + 1); + mark_object (kb->system_key_syms, depth + 1); + mark_object (kb->Vdefault_minibuffer_frame, depth + 1); + mark_object (kb->echo_string, depth + 1); } { struct input_event *event; @@ -11455,11 +11456,11 @@ event = kbd_buffer; if (event->kind != SELECTION_REQUEST_EVENT) { - mark_object (event->x); - mark_object (event->y); + mark_object (event->x, depth + 1); + mark_object (event->y, depth + 1); } - mark_object (event->frame_or_window); - mark_object (event->arg); + mark_object (event->frame_or_window, depth + 1); + mark_object (event->arg, depth + 1); } } } --- orig/src/lisp.h +++ mod/src/lisp.h @@ -2440,9 +2440,12 @@ extern void memory_full P_ ((void)); extern void buffer_memory_full P_ ((void)); extern int survives_gc_p P_ ((Lisp_Object)); -extern void mark_object P_ ((Lisp_Object)); +extern void mark_object P_ ((Lisp_Object, unsigned)); +extern void mark_kboards P_ ((unsigned)); +extern void mark_backtrace P_ ((void)); extern Lisp_Object Vpurify_flag; extern Lisp_Object Vmemory_full; +extern int message_enable_multibyte; EXFUN (Fcons, 2); EXFUN (list2, 2); EXFUN (list3, 3); ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-27 19:40 ` Florian Weimer @ 2004-09-27 19:52 ` Stefan Monnier 2004-09-27 20:36 ` Florian Weimer 2004-09-27 20:48 ` Simon Josefsson 2004-09-28 15:20 ` Richard Stallman 1 sibling, 2 replies; 15+ messages in thread From: Stefan Monnier @ 2004-09-27 19:52 UTC (permalink / raw) Cc: emacs-devel > while they are traversed. Unfortunately, no major leak turned up, > just a few thousand cons cells which can't cause the massive leak I > see (in the order of dozens of megabytes after entering/exiting a > large NNTP group with Gnus). I think it would be worthwhile to start with the basic info returned by `garbage-collect' and such stuff. See `memory-usage.el' below. [ I thought you already tried that, but the few lines above make me think that you don't even know whether the extra MBs are made up of cons cells, or strings, or non-elisp-data, or ...] > Is there a method to determine the (approximate) size of a buffer? See buffer-size-bytes below. > Are there any other objects that can change their size after > allocation? (I'm pretty sure that there are no additional Lisp > objects allocated, but maybe an existing object grows without bounds.) Yes. Things like the specpdl "stack", the matrices used in display, maybe the text-property-tree nodes, ... Stefan ;;; memory-usage.el --- Analyze the memory usage of Emacs in various ways ;; Copyright (C) 2002, 2004 Free Software Foundation, Inc. ;; Author: Stefan Monnier <monnier@cs.yale.edu> ;; Keywords: maint ;; This file is free software; you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation; either version 2, or (at your option) ;; any later version. ;; This file is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; You should have received a copy of the GNU General Public License ;; along with GNU Emacs; see the file COPYING. If not, write to ;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330, ;; Boston, MA 02111-1307, USA. ;;; Commentary: ;; ;;; Code: (defun buffer-size-bytes (b) "Return total number of bytes in the buffer contents." (with-current-buffer b (save-restriction (widen) (- (position-bytes (point-max)) (position-bytes (point-min)))))) (defun buffer-gap-bytes (b) "Return total number of bytes in the buffer gap." (with-current-buffer b (gap-size))) (defun buffer-total-bytes (b) "Return total number of ralloc bytes used by buffer." (with-current-buffer b (save-restriction (widen) (+ (position-bytes (point-max)) (- (position-bytes (point-min))) (gap-size))))) ;;;###autoload (defun memory-usage () "List all buffers and their memory usage." (interactive) (pop-to-buffer (get-buffer-create "*Buffer Details*")) (erase-buffer) (let* ((bufs (buffer-list)) (num (length bufs)) (gc-stats (garbage-collect)) (conses (nth 0 gc-stats)) (symbols (nth 1 gc-stats)) (markers (nth 2 gc-stats)) (strings (nth 3 gc-stats)) (vectors (nth 4 gc-stats)) (floats (nth 5 gc-stats)) (intervals (nth 6 gc-stats))) (insert (format "Garbage collection stats:\n%s\n\n =>" gc-stats)) (insert (format "\t%d bytes in cons cells\n" (* 8 (+ (car conses) (cdr conses))))) (insert (format "\t%d bytes in symbols\n" (* 24 (+ (car symbols) (cdr symbols))))) (insert (format "\t%d bytes in markers\n" (* 20 (+ (car markers) (cdr markers))))) (insert (format "\t%d bytes of string chars\n" strings)) (insert (format "\t%d bytes of vector slots\n" (* 4 vectors))) (insert (format "\t%d bytes in floats\n" (* 12 (+ (car floats) (cdr floats))))) (insert (format "\t%d bytes in intervals\n" (* 28 (+ (car intervals) (cdr intervals))))) (insert (format "\nTotal bytes in lisp objects (not counting string and vector headers): %d\n\n" (+ (* 8 (+ (car conses) (cdr conses))) (* 24 (+ (car symbols) (cdr symbols))) (* 20 (+ (car markers) (cdr markers))) strings vectors (* 12 (+ (car floats) (cdr floats))) (* 28 (+ (car intervals) (cdr intervals)))))) (insert (format "Buffer ralloc memory usage:\n%d buffers\n%d bytes total (%d in gaps)\n" num (apply #'+ (mapcar #'buffer-total-bytes bufs)) (apply #'+ (mapcar #'buffer-gap-bytes bufs)))) (insert (format "%10s\t%s\t%s\n\n" "Size" "Gap" "Name")) (insert (mapconcat (lambda (b) (format "%10d\t%s\t%s" (buffer-size-bytes b) (buffer-gap-bytes b) (buffer-name b))) (sort bufs (lambda (b1 b2) (> (buffer-size-bytes b1) (buffer-size-bytes b2)))) "\n")) (insert "\n")) (goto-char (point-min))) (provide 'memory-usage) ;; arch-tag: 04e012f0-3c59-4319-8d1a-e86204671ec5 ;;; memory-usage.el ends here ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-27 19:52 ` Stefan Monnier @ 2004-09-27 20:36 ` Florian Weimer 2004-09-27 20:49 ` Stefan Monnier 2004-09-27 20:48 ` Simon Josefsson 1 sibling, 1 reply; 15+ messages in thread From: Florian Weimer @ 2004-09-27 20:36 UTC (permalink / raw) Cc: emacs-devel * Stefan Monnier: > I think it would be worthwhile to start with the basic info returned by > `garbage-collect' and such stuff. See `memory-usage.el' below. > [ I thought you already tried that, but the few lines above make me think > that you don't even know whether the extra MBs are made up of cons cells, or > strings, or non-elisp-data, or ...] Others already did this. But thanks anyway, I repeated the experiment: The numbers indeed lack any distinct increase in magnitude. 8-( >> Are there any other objects that can change their size after >> allocation? (I'm pretty sure that there are no additional Lisp >> objects allocated, but maybe an existing object grows without bounds.) > > Yes. Things like the specpdl "stack", the matrices used in display, maybe > the text-property-tree nodes, ... Hmm, I'm going to gather a few more stats. (And running Emacs under valgrind could prove helpful because I could determine if the low-level allocator leaks memory.) ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-27 20:36 ` Florian Weimer @ 2004-09-27 20:49 ` Stefan Monnier 0 siblings, 0 replies; 15+ messages in thread From: Stefan Monnier @ 2004-09-27 20:49 UTC (permalink / raw) Cc: emacs-devel >> I think it would be worthwhile to start with the basic info returned by >> `garbage-collect' and such stuff. See `memory-usage.el' below. >> [ I thought you already tried that, but the few lines above make me think >> that you don't even know whether the extra MBs are made up of cons cells, or >> strings, or non-elisp-data, or ...] > Others already did this. But thanks anyway, I repeated the > experiment: The numbers indeed lack any distinct increase in > magnitude. 8-( Well, that means that the increase is apparently elsewhere, which is good because it already tells you that hacking on the gc_marking code probably won't help you. Now, that doesn't rule out all of alloc.c :-( >>> Are there any other objects that can change their size after >>> allocation? (I'm pretty sure that there are no additional Lisp >>> objects allocated, but maybe an existing object grows without bounds.) >> >> Yes. Things like the specpdl "stack", the matrices used in display, maybe >> the text-property-tree nodes, ... > Hmm, I'm going to gather a few more stats. > (And running Emacs under valgrind could prove helpful because I could > determine if the low-level allocator leaks memory.) Another thing you might want to try is: - check /proc/$$/maps before and after the memory growth. - based on that, infer the memory range where the new (tho supposedly dead) data is located. - from GDB, peek at that memory range to see if you recognize the kind of data that's there. Stefan ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-27 19:52 ` Stefan Monnier 2004-09-27 20:36 ` Florian Weimer @ 2004-09-27 20:48 ` Simon Josefsson 1 sibling, 0 replies; 15+ messages in thread From: Simon Josefsson @ 2004-09-27 20:48 UTC (permalink / raw) Stefan Monnier <monnier@iro.umontreal.ca> writes: >> while they are traversed. Unfortunately, no major leak turned up, >> just a few thousand cons cells which can't cause the massive leak I >> see (in the order of dozens of megabytes after entering/exiting a >> large NNTP group with Gnus). > > I think it would be worthwhile to start with the basic info returned by > `garbage-collect' and such stuff. See `memory-usage.el' below. > [ I thought you already tried that, but the few lines above make me think > that you don't even know whether the extra MBs are made up of cons cells, or > strings, or non-elisp-data, or ...] Thanks, memory-usage seem useful. The memory growth caused by Gnus seem to escape it though. This is a fairly fresh emacs (I'm using Gnus, programming C and reading documentation, but nothing fancy): 20807 jas 15 0 365m 185m 17m S 2.7 18.3 89:12.64 emacs But there is no indication of where the memory is used (see below). Perhaps the "top" statistics aren't reliable, but the fact remains that the sluggishness I feel appear to be correlated to high memory usage according to "top". Hm. Is it possible to use 'gdb' to find out where all memory goes? Seems like it should have the capability to do it, at least. (FWIW, the above emacs is not sluggish.) Garbage collection stats: ((1130529 . 100249) (60991 . 170) (8108 . 5450) 3508516 1049895 (322 . 551) (34955 . 28764) (143989 . 25103)) => 9846224 bytes in cons cells 1467864 bytes in symbols 271160 bytes in markers 3508516 bytes of string chars 4199580 bytes of vector slots 10476 bytes in floats 1784132 bytes in intervals Total bytes in lisp objects (not counting string and vector headers): 17938267 Buffer ralloc memory usage: 147 buffers 5564167 bytes total (66581 in gaps) Size Gap Name 1843610 2000 *DOC* 524443 2000 snprintf_2.2.tar.gz 467401 2000 *Messages* 460876 2000 *Gnus Backlog* ... ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-27 19:40 ` Florian Weimer 2004-09-27 19:52 ` Stefan Monnier @ 2004-09-28 15:20 ` Richard Stallman 2004-09-28 21:00 ` Florian Weimer 2004-09-28 21:40 ` Kim F. Storm 1 sibling, 2 replies; 15+ messages in thread From: Richard Stallman @ 2004-09-28 15:20 UTC (permalink / raw) Cc: emacs-devel It might be interesting to keep track of malloc'ing of large blocks of memory, recording the last N large blocks allocated in a table. Or perhaps only those that do not hold conses, miscs, small strings, etc. This way you might be able to find some large blocks and then examine them to see what data is in them. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-28 15:20 ` Richard Stallman @ 2004-09-28 21:00 ` Florian Weimer 2004-09-28 21:51 ` Florian Weimer ` (2 more replies) 2004-09-28 21:40 ` Kim F. Storm 1 sibling, 3 replies; 15+ messages in thread From: Florian Weimer @ 2004-09-28 21:00 UTC (permalink / raw) Cc: emacs-devel * Richard Stallman: > It might be interesting to keep track of malloc'ing of large blocks of > memory, recording the last N large blocks allocated in a table. > Or perhaps only those that do not hold conses, miscs, small strings, etc. > This way you might be able to find some large blocks and then > examine them to see what data is in them. The approach which I have used with great success in the past is to use GNU libc's malloc() hooks to print complete malloc()/realloc()/free() traces. The crucial point is to include information from backtrace() (because otherwise, we only see calls to xmalloc()). A short Python (or whatever) script is then used to discover blocks which are never freed, sums up the leakage by backtrace, and uses addr2line to print a human-readable backtrace. There is a tool, called valgrind, that uses dynamic recompilation of x86 code to inject such tracing code without recompilation and relinking. It can detect memory leaks, but it catches a wide range of programming mistakes, too. Unfortunately, a dumped Emacs does not run under valgrind, so I could use it. Therefore, I'm going to post my enhanced alloc tracing patch in a separate message for future reference. Now to the interesting part. The worst offender is decode-coding-region: * 42212224 dump_hook_malloc /home/fw/src/gnu/emacs-memory-dump/src/alloc.c:5740 ?? ??:0 coding_allocate_composition_data /home/fw/src/gnu/emacs-memory-dump/src/coding.c:1699 code_convert_region /home/fw/src/gnu/emacs-memory-dump/src/coding.c:5650 code_convert_region1 /home/fw/src/gnu/emacs-memory-dump/src/coding.c:6966 Fdecode_coding_region /home/fw/src/gnu/emacs-memory-dump/src/coding.c:6983 Ffuncall /home/fw/src/gnu/emacs-memory-dump/src/eval.c:2743 Fbyte_code /home/fw/src/gnu/emacs-memory-dump/src/bytecode.c:686 funcall_lambda /home/fw/src/gnu/emacs-memory-dump/src/eval.c:2928 Ffuncall /home/fw/src/gnu/emacs-memory-dump/src/eval.c:2803 Fbyte_code /home/fw/src/gnu/emacs-memory-dump/src/bytecode.c:686 Feval /home/fw/src/gnu/emacs-memory-dump/src/eval.c:2097 Fcondition_case /home/fw/src/gnu/emacs-memory-dump/src/eval.c:1293 Fbyte_code /home/fw/src/gnu/emacs-memory-dump/src/bytecode.c:864 Feval /home/fw/src/gnu/emacs-memory-dump/src/eval.c:2097 Fcondition_case /home/fw/src/gnu/emacs-memory-dump/src/eval.c:1293 Fbyte_code /home/fw/src/gnu/emacs-memory-dump/src/bytecode.c:864 funcall_lambda /home/fw/src/gnu/emacs-memory-dump/src/eval.c:2928 Ffuncall /home/fw/src/gnu/emacs-memory-dump/src/eval.c:2803 It turns out that we are returning from code_convert_region() without freeing the composition data under certain circumstances. This bug is not new, it has been present since code_convert_region() started allocating data using xmalloc() in late 1999. The recent internationalization efforts just uncovered it. The patch below is for illustration only. It fixes the memory leak, but maybe other cleanups are missing on this path. 2004-09-28 Florian Weimer <fw@deneb.enyo.de> * coding.c (code_convert_region): Free composition data to fix memory leak. --- emacs-upstream/src/coding.c 2004-09-27 21:32:39.000000000 +0200 +++ emacs-memory-dump/src/coding.c 2004-09-28 22:26:13.000000000 +0200 @@ -5667,6 +5667,7 @@ if (!replace) /* We must record and adjust for this new text now. */ adjust_after_insert (from, from_byte_orig, to, to_byte_orig, len); + coding_free_composition_data (coding); return 0; } (Disclaimer: I haven't got papers on file, but there is just one way to make this change, so I doubt it is copyrightable.) ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-28 21:00 ` Florian Weimer @ 2004-09-28 21:51 ` Florian Weimer 2004-09-29 16:39 ` Richard Stallman 2004-09-29 23:51 ` Kenichi Handa 2 siblings, 0 replies; 15+ messages in thread From: Florian Weimer @ 2004-09-28 21:51 UTC (permalink / raw) Here's the updated patch for future reference. --- emacs-upstream/src/find-leak.py 1970-01-01 01:00:00.000000000 +0100 +++ emacs-memory-dump/src/find-leak.py 2004-09-28 23:35:37.000000000 +0200 @@ -0,0 +1,108 @@ +# A patched Emacs writes malloc() traces to standard error after user +# has invoked enable-malloc-tracing. This script processes the output +# and produces some statistics. It finds non-freed memory blocks and +# groups them by the backtrace at the point of allocation. After +# that, the leaking backtraces are sorted by decreasing leakage, and +# the addresses are resolved using addr2line. +# +# You should pass this script the name of the file to which Emacs +# wrote the malloc trace on the command line. The file might contain +# some garbled lines with multiple "@" signs (possibly due to some +# obscure reentrance problems that have not yet been solved). Delete +# them if this script runs into them. +# +# It is a good idea to repeat the action that triggers the memory leak +# multiple times (so that memory-limit grows by a substantial value), +# then invoke garbage-collect (to free as much memory as possible), +# and exit from Emacs. Some Lisp objects will not be freed (and show +# up in the leakage statistics), but the real memory leak will be +# substantially larger and easy to spot in the sorted output. +# +# Note that if your system's libc library has been compiled with frame +# pointer omission, the output won't contain usable information. You +# must use the debugging version of libc. (On Debian systems, it's +# available in the libc6-dbg package, and you have to set the +# LD_LIBRARY_PATH environment variable to "/usr/lib/debug".) +# +# Note that this approach won't be very helpful if the leak is caused +# by unused, but still-referenced Lisp data. Use garbage-collect-dump +# (in the patched Emacs) for that and hope for the best. Its output +# is more structured and might reveal which Lisp variable is the +# culprit. + +import sys, os + +def convert_allocators(allocs): + l = [] + for a in allocs: + if a <> '(nil)': + l.append(eval(a)) + else: + l.append(0) + return tuple(l) + +def unfreed_objects(file): + objects = {} + + for line in file.readlines(): + fields = line.split(' ') + if fields[0] <> '@': + continue + + if fields[1] == 'A': + address = eval(fields[2]) + size = int(fields[3]) + if fields[4] <> ':': + raise ValueError + objects[address] = (size, convert_allocators(fields[5:])) + + elif fields[1] == 'R': + address = fields[2] + if address == '(nil)': + address = 0 + else: + address = eval(address) + + new_address = eval(fields[3]) + size = int(fields[4]) + if fields[5] <> ':': + raise ValueError + if objects.has_key(address): + del objects[address] + objects[new_address] = (size, convert_allocators(fields[6:])) + + elif fields[1] == 'F': + address = eval(fields[2]) + if objects.has_key(address): + del objects[address] + + return objects + +def leaking_paths(objects): + paths = {} + for (size, path) in objects.values(): + paths[path] = paths.get(path, 0) + size + return paths + +objects = unfreed_objects(open(sys.argv[1])) +paths = leaking_paths(objects) + +resolve_cache = {} +def resolve(addr): + if resolve_cache.has_key(addr): + return resolve_cache[addr] + address = hex(p) + lines = os.popen("addr2line -e ./emacs -f " + address, "r").readlines() + result = tuple("".join(lines).split('\n')) + resolve_cache[addr] = result + return result + +l = map(lambda (path, size): (-size, path), paths.items()) +l.sort() +for (size, path) in l: + size = -size + print '*', size + for p in path: + print " " + " ".join(resolve(p)) + +# arch-tag: 38cf01b5-1b4d-405e-b324-0f79a146826f --- emacs-upstream/src/lisp.h 2004-09-27 21:32:39.000000000 +0200 +++ emacs-memory-dump/src/lisp.h 2004-09-28 20:49:45.000000000 +0200 @@ -2440,9 +2440,12 @@ extern void memory_full P_ ((void)); extern void buffer_memory_full P_ ((void)); extern int survives_gc_p P_ ((Lisp_Object)); -extern void mark_object P_ ((Lisp_Object)); +extern void mark_object P_ ((Lisp_Object, unsigned)); +extern void mark_kboards P_ ((unsigned)); +extern void mark_backtrace P_ ((void)); extern Lisp_Object Vpurify_flag; extern Lisp_Object Vmemory_full; +extern int message_enable_multibyte; EXFUN (Fcons, 2); EXFUN (list2, 2); EXFUN (list3, 3); --- emacs-upstream/src/alloc.c 2004-09-27 21:33:09.000000000 +0200 +++ emacs-memory-dump/src/alloc.c 2004-09-28 21:10:38.000000000 +0200 @@ -258,12 +258,15 @@ Lisp_Object Vgc_elapsed; /* accumulated elapsed time in GC */ EMACS_INT gcs_done; /* accumulated GCs */ -static void mark_buffer P_ ((Lisp_Object)); -extern void mark_kboards P_ ((void)); -extern void mark_backtrace P_ ((void)); +/* If non-zero, dump objects to stderr while they are marked. */ +static int do_dump = 0; + +static void dump_marker_section (char *name); + +static void mark_buffer P_ ((Lisp_Object, unsigned)); static void gc_sweep P_ ((void)); -static void mark_glyph_matrix P_ ((struct glyph_matrix *)); -static void mark_face_cache P_ ((struct face_cache *)); +static void mark_glyph_matrix P_ ((struct glyph_matrix *, unsigned)); +static void mark_face_cache P_ ((struct face_cache *, unsigned)); #ifdef HAVE_WINDOW_SYSTEM static void mark_image P_ ((struct image *)); @@ -275,8 +278,6 @@ static void free_large_strings P_ ((void)); static void sweep_strings P_ ((void)); -extern int message_enable_multibyte; - /* When scanning the C stack for live Lisp objects, Emacs keeps track of what memory allocated via lisp_malloc is intended for what purpose. This enumeration specifies the type of memory. */ @@ -389,8 +390,8 @@ static int live_symbol_p P_ ((struct mem_node *, void *)); static int live_float_p P_ ((struct mem_node *, void *)); static int live_misc_p P_ ((struct mem_node *, void *)); -static void mark_maybe_object P_ ((Lisp_Object)); -static void mark_memory P_ ((void *, void *)); +static void mark_maybe_object P_ ((Lisp_Object, unsigned)); +static void mark_memory P_ ((void *, void *, unsigned)); static void mem_init P_ ((void)); static struct mem_node *mem_insert P_ ((void *, void *, enum mem_type)); static void mem_insert_fixup P_ ((struct mem_node *)); @@ -1217,7 +1218,7 @@ { eassert (!i->gcmarkbit); /* Intervals are never shared. */ i->gcmarkbit = 1; - mark_object (i->plist); + mark_object (i->plist, 0); } @@ -3688,8 +3689,9 @@ /* Mark OBJ if we can prove it's a Lisp_Object. */ static INLINE void -mark_maybe_object (obj) +mark_maybe_object (obj, depth) Lisp_Object obj; + unsigned depth; { void *po = (void *) XPNTR (obj); struct mem_node *m = mem_find (po); @@ -3743,7 +3745,7 @@ zombies[nzombies] = obj; ++nzombies; #endif - mark_object (obj); + mark_object (obj, depth + 1); } } } @@ -3753,8 +3755,9 @@ marked. */ static INLINE void -mark_maybe_pointer (p) +mark_maybe_pointer (p, depth) void *p; + unsigned depth; { struct mem_node *m; @@ -3824,7 +3827,7 @@ } if (!GC_NILP (obj)) - mark_object (obj); + mark_object (obj, depth + 1); } } @@ -3832,8 +3835,9 @@ /* Mark Lisp objects referenced from the address range START..END. */ static void -mark_memory (start, end) +mark_memory (start, end, depth) void *start, *end; + unsigned depth; { Lisp_Object *p; void **pp; @@ -3853,7 +3857,7 @@ /* Mark Lisp_Objects. */ for (p = (Lisp_Object *) start; (void *) p < end; ++p) - mark_maybe_object (*p); + mark_maybe_object (*p, depth + 1); /* Mark Lisp data pointed to. This is necessary because, in some situations, the C compiler optimizes Lisp objects away, so that @@ -4098,7 +4102,7 @@ #endif #endif for (i = 0; i < sizeof (Lisp_Object); i += GC_LISP_OBJECT_ALIGNMENT) - mark_memory ((char *) stack_base + i, end); + mark_memory ((char *) stack_base + i, end, 1); #if GC_MARK_STACK == GC_MARK_STACK_CHECK_GCPROS check_gcpros (); @@ -4455,14 +4459,14 @@ /* Mark all the special slots that serve as the roots of accessibility. */ for (i = 0; i < staticidx; i++) - mark_object (*staticvec[i]); + mark_object (*staticvec[i], 1); for (bind = specpdl; bind != specpdl_ptr; bind++) { - mark_object (bind->symbol); - mark_object (bind->old_value); + mark_object (bind->symbol, 1); + mark_object (bind->old_value, 1); } - mark_kboards (); + mark_kboards (1); #ifdef USE_GTK { @@ -4483,18 +4487,22 @@ } #endif + dump_marker_section ("Begin marking byte stack."); mark_byte_stack (); + dump_marker_section ("Begin marking byte stack."); for (catch = catchlist; catch; catch = catch->next) { - mark_object (catch->tag); - mark_object (catch->val); + mark_object (catch->tag, 1); + mark_object (catch->val, 1); } for (handler = handlerlist; handler; handler = handler->next) { - mark_object (handler->handler); - mark_object (handler->var); + mark_object (handler->handler, 1); + mark_object (handler->var, 1); } + dump_marker_section ("Begin marking backtrace."); mark_backtrace (); + dump_marker_section ("End marking backtrace."); #if GC_MARK_STACK == GC_USE_GCPROS_CHECK_ZOMBIES mark_stack (); @@ -4542,7 +4550,7 @@ } /* Now that we have stripped the elements that need not be in the undo_list any more, we can finally mark the list. */ - mark_object (nextb->undo_list); + mark_object (nextb->undo_list, 1); nextb = nextb->next; } @@ -4630,13 +4638,24 @@ return Flist (sizeof total / sizeof *total, total); } +DEFUN ("garbage-collect-dump", Fgarbage_collect_dump, Sgarbage_collect_dump, 0, 0, "", + doc: /* Run garbage collection and dump objects. */) + () +{ + ++do_dump; + dump_marker_section ("Begin garbage collection."); + Fgarbage_collect (); + dump_marker_section ("End garbage collection."); + --do_dump; +} /* Mark Lisp objects in glyph matrix MATRIX. Currently the only interesting objects referenced from glyphs are strings. */ static void -mark_glyph_matrix (matrix) +mark_glyph_matrix (matrix, depth) struct glyph_matrix *matrix; + unsigned depth; { struct glyph_row *row = matrix->rows; struct glyph_row *end = row + matrix->nrows; @@ -4653,7 +4672,7 @@ for (; glyph < end_glyph; ++glyph) if (GC_STRINGP (glyph->object) && !STRING_MARKED_P (XSTRING (glyph->object))) - mark_object (glyph->object); + mark_object (glyph->object, depth + 1); } } } @@ -4662,8 +4681,9 @@ /* Mark Lisp faces in the face cache C. */ static void -mark_face_cache (c) +mark_face_cache (c, depth) struct face_cache *c; + unsigned depth; { if (c) { @@ -4675,7 +4695,7 @@ if (face) { for (j = 0; j < LFACE_VECTOR_SIZE; ++j) - mark_object (face->lface[j]); + mark_object (face->lface[j], depth + 1); } } } @@ -4690,10 +4710,10 @@ mark_image (img) struct image *img; { - mark_object (img->spec); + mark_object (img->spec, 1); if (!NILP (img->data.lisp_val)) - mark_object (img->data.lisp_val); + mark_object (img->data.lisp_val, 1); } @@ -4725,9 +4745,84 @@ Normally this is zero and the check never goes off. */ int mark_object_loop_halt; +/* For memory debugging: dumps the string in human-readable form to + stderr. */ +static void +dump_string (char *data, unsigned length) +{ + unsigned i; + + for (i = 0; i < length; ++i) + { + char c = data[i]; + if (c >= 32 && c <= 126) + fputc (c, stderr); + else if (c == '"' || c == '\\') + { + fputc ('\\', stderr); + fputc (c, stderr); + } + else if (c == '\n') + fputs ("\\n", stderr); + else if (c == '\t') + fputs ("\\t", stderr); + else if (c == '\r') + fputs ("\\r", stderr); + else + fprintf (stderr, "\\%03o", (unsigned)(unsigned char) c); + } +} + +/* For memory debugging: prints OBJ of TYPE, at nesting level + DEPTH to stderr. */ +static void +dump_object (char *type, unsigned depth, Lisp_Object obj, int marked) +{ + unsigned i; + + if (!do_dump) + return; + + fprintf (stderr, "[%u] %s (%p)%s\n", + depth, type, (void *)obj, marked ? " *" : ""); +} + +static void +dump_object_int (unsigned depth, Lisp_Object obj) +{ + if (!do_dump) + return; + + fprintf (stderr, "[%u] INT %d\n", depth, XINT(obj)); +} + +static void +dump_marker_section(char *name) +{ + if (!do_dump) + return; + + fprintf (stderr, "%s\n", name); +} + +/* For memory debugging: prints OBJ of TYPE, with NAME, at nesting + level DEPTH to stderr. */ +static void +dump_object_name (char *type, unsigned depth, Lisp_Object obj, struct Lisp_String* name, int marked) +{ + if (!do_dump) + return; + + fprintf (stderr, "[%u] %s (%p) \"", depth, type, (void *)obj); + dump_string (name->data, name->size & ~ARRAY_MARK_FLAG); + fprintf (stderr, "\"%s\n", marked ? " *" : ""); +} + + void -mark_object (arg) +mark_object (arg, depth) Lisp_Object arg; + unsigned depth; { register Lisp_Object obj = arg; #ifdef GC_CHECK_MARKED_OBJECTS @@ -4792,6 +4887,7 @@ CHECK_ALLOCATED_AND_LIVE (live_string_p); MARK_INTERVAL_TREE (ptr->intervals); MARK_STRING (ptr); + dump_object_name ("STRING", depth, obj, ptr, ptr->size & ARRAY_MARK_FLAG); #ifdef GC_CHECK_STRING_BYTES /* Check that the string size recorded in the string is the same as the one recorded in the sdata structure. */ @@ -4811,6 +4907,8 @@ if (GC_BUFFERP (obj)) { + dump_object ("BUFFER", depth, obj, VECTOR_MARKED_P (XBUFFER (obj))); + if (!VECTOR_MARKED_P (XBUFFER (obj))) { #ifdef GC_CHECK_MARKED_OBJECTS @@ -4823,7 +4921,7 @@ abort (); } #endif /* GC_CHECK_MARKED_OBJECTS */ - mark_buffer (obj); + mark_buffer (obj, depth + 1); } } else if (GC_SUBRP (obj)) @@ -4837,6 +4935,8 @@ register EMACS_INT size = ptr->size; register int i; + dump_object ("COMPILED", depth, obj, VECTOR_MARKED_P (ptr)); + if (VECTOR_MARKED_P (ptr)) break; /* Already marked */ @@ -4846,7 +4946,7 @@ for (i = 0; i < size; i++) /* and then mark its elements */ { if (i != COMPILED_CONSTANTS) - mark_object (ptr->contents[i]); + mark_object (ptr->contents[i], depth + 1); } obj = ptr->contents[COMPILED_CONSTANTS]; goto loop; @@ -4855,40 +4955,48 @@ { register struct frame *ptr = XFRAME (obj); + dump_object ("FRAME", depth, obj, VECTOR_MARKED_P (ptr)); + if (VECTOR_MARKED_P (ptr)) break; /* Already marked */ + VECTOR_MARK (ptr); /* Else mark it */ CHECK_LIVE (live_vector_p); - mark_object (ptr->name); - mark_object (ptr->icon_name); - mark_object (ptr->title); - mark_object (ptr->focus_frame); - mark_object (ptr->selected_window); - mark_object (ptr->minibuffer_window); - mark_object (ptr->param_alist); - mark_object (ptr->scroll_bars); - mark_object (ptr->condemned_scroll_bars); - mark_object (ptr->menu_bar_items); - mark_object (ptr->face_alist); - mark_object (ptr->menu_bar_vector); - mark_object (ptr->buffer_predicate); - mark_object (ptr->buffer_list); - mark_object (ptr->menu_bar_window); - mark_object (ptr->tool_bar_window); - mark_face_cache (ptr->face_cache); + mark_object (ptr->name, depth + 1); + mark_object (ptr->icon_name, depth + 1); + mark_object (ptr->title, depth + 1); + mark_object (ptr->focus_frame, depth + 1); + mark_object (ptr->selected_window, depth + 1); + mark_object (ptr->minibuffer_window, depth + 1); + mark_object (ptr->param_alist, depth + 1); + mark_object (ptr->scroll_bars, depth + 1); + mark_object (ptr->condemned_scroll_bars, depth + 1); + mark_object (ptr->menu_bar_items, depth + 1); + mark_object (ptr->face_alist, depth + 1); + mark_object (ptr->menu_bar_vector, depth + 1); + mark_object (ptr->buffer_predicate, depth + 1); + mark_object (ptr->buffer_list, depth + 1); + mark_object (ptr->menu_bar_window, depth + 1); + mark_object (ptr->tool_bar_window, depth + 1); + mark_face_cache (ptr->face_cache, depth + 1); #ifdef HAVE_WINDOW_SYSTEM + dump_marker_section ("Begin marking FRAME images."); mark_image_cache (ptr); - mark_object (ptr->tool_bar_items); - mark_object (ptr->desired_tool_bar_string); - mark_object (ptr->current_tool_bar_string); + dump_marker_section ("End marking FRAME images."); + mark_object (ptr->tool_bar_items, depth + 1); + mark_object (ptr->desired_tool_bar_string, depth + 1); + mark_object (ptr->current_tool_bar_string, depth + 1); #endif /* HAVE_WINDOW_SYSTEM */ } else if (GC_BOOL_VECTOR_P (obj)) { register struct Lisp_Vector *ptr = XVECTOR (obj); + dump_object ("BOOL_VECTOR", depth, obj, VECTOR_MARKED_P (ptr)); + if (VECTOR_MARKED_P (ptr)) break; /* Already marked */ + CHECK_LIVE (live_vector_p); VECTOR_MARK (ptr); /* Else mark it */ } @@ -4898,6 +5006,8 @@ struct window *w = XWINDOW (obj); register int i; + dump_object ("WINDOW", depth, obj, VECTOR_MARKED_P (ptr)); + /* Stop if already marked. */ if (VECTOR_MARKED_P (ptr)) break; @@ -4911,7 +5021,7 @@ for (i = 0; (char *) &ptr->contents[i] < (char *) &w->current_matrix; i++) - mark_object (ptr->contents[i]); + mark_object (ptr->contents[i], depth + 1); /* Mark glyphs for leaf windows. Marking window matrices is sufficient because frame matrices use the same glyph @@ -4920,14 +5030,16 @@ && NILP (w->vchild) && w->current_matrix) { - mark_glyph_matrix (w->current_matrix); - mark_glyph_matrix (w->desired_matrix); + mark_glyph_matrix (w->current_matrix, depth + 1); + mark_glyph_matrix (w->desired_matrix, depth + 1); } } else if (GC_HASH_TABLE_P (obj)) { struct Lisp_Hash_Table *h = XHASH_TABLE (obj); + dump_object ("HASH_TABLE", depth, obj, VECTOR_MARKED_P (h)); + /* Stop if already marked. */ if (VECTOR_MARKED_P (h)) break; @@ -4941,20 +5053,20 @@ Being in the next_weak chain should not keep the hash table alive. No need to mark `count' since it is an integer. */ - mark_object (h->test); - mark_object (h->weak); - mark_object (h->rehash_size); - mark_object (h->rehash_threshold); - mark_object (h->hash); - mark_object (h->next); - mark_object (h->index); - mark_object (h->user_hash_function); - mark_object (h->user_cmp_function); + mark_object (h->test, depth + 1); + mark_object (h->weak, depth + 1); + mark_object (h->rehash_size, depth + 1); + mark_object (h->rehash_threshold, depth + 1); + mark_object (h->hash, depth + 1); + mark_object (h->next, depth + 1); + mark_object (h->index, depth + 1); + mark_object (h->user_hash_function, depth + 1); + mark_object (h->user_cmp_function, depth + 1); /* If hash table is not weak, mark all keys and values. For weak tables, mark only the vector. */ if (GC_NILP (h->weak)) - mark_object (h->key_and_value); + mark_object (h->key_and_value, depth + 1); else VECTOR_MARK (XVECTOR (h->key_and_value)); } @@ -4964,6 +5076,8 @@ register EMACS_INT size = ptr->size; register int i; + dump_object ("VECTOR", depth, obj, VECTOR_MARKED_P (ptr)); + if (VECTOR_MARKED_P (ptr)) break; /* Already marked */ CHECK_LIVE (live_vector_p); VECTOR_MARK (ptr); /* Else mark it */ @@ -4971,7 +5085,7 @@ size &= PSEUDOVECTOR_SIZE_MASK; for (i = 0; i < size; i++) /* and then mark its elements */ - mark_object (ptr->contents[i]); + mark_object (ptr->contents[i], depth + 1); } break; @@ -4980,12 +5094,15 @@ register struct Lisp_Symbol *ptr = XSYMBOL (obj); struct Lisp_Symbol *ptrx; + dump_object_name ("SYMBOL", depth, obj, XSTRING (ptr->xname), ptr->gcmarkbit); + if (ptr->gcmarkbit) break; + CHECK_ALLOCATED_AND_LIVE (live_symbol_p); ptr->gcmarkbit = 1; - mark_object (ptr->value); - mark_object (ptr->function); - mark_object (ptr->plist); + mark_object (ptr->value, depth + 1); + mark_object (ptr->function, depth + 1); + mark_object (ptr->plist, depth + 1); if (!PURE_POINTER_P (XSTRING (ptr->xname))) MARK_STRING (XSTRING (ptr->xname)); @@ -5006,6 +5123,7 @@ case Lisp_Misc: CHECK_ALLOCATED_AND_LIVE (live_misc_p); + dump_object ("MISC", depth, obj, XMARKER (obj)->gcmarkbit); if (XMARKER (obj)->gcmarkbit) break; XMARKER (obj)->gcmarkbit = 1; @@ -5023,9 +5141,9 @@ obj = ptr->realvalue; goto loop; } - mark_object (ptr->realvalue); - mark_object (ptr->buffer); - mark_object (ptr->frame); + mark_object (ptr->realvalue, depth + 1); + mark_object (ptr->buffer, depth + 1); + mark_object (ptr->frame, depth + 1); obj = ptr->cdr; goto loop; } @@ -5058,18 +5176,17 @@ Lisp_Object *p = (Lisp_Object *) ptr->pointer; int nelt; for (nelt = ptr->integer; nelt > 0; nelt--, p++) - mark_maybe_object (*p); + mark_maybe_object (*p, depth + 1); } } #endif break; - case Lisp_Misc_Overlay: { struct Lisp_Overlay *ptr = XOVERLAY (obj); - mark_object (ptr->start); - mark_object (ptr->end); - mark_object (ptr->plist); + mark_object (ptr->start, depth + 1); + mark_object (ptr->end, depth + 1); + mark_object (ptr->plist, depth + 1); if (ptr->next) { XSETMISC (obj, ptr->next); @@ -5086,30 +5203,23 @@ case Lisp_Cons: { register struct Lisp_Cons *ptr = XCONS (obj); + dump_object ("CONS", depth, obj, CONS_MARKED_P (ptr)); if (CONS_MARKED_P (ptr)) break; CHECK_ALLOCATED_AND_LIVE (live_cons_p); CONS_MARK (ptr); - /* If the cdr is nil, avoid recursion for the car. */ - if (EQ (ptr->cdr, Qnil)) - { - obj = ptr->car; - cdr_count = 0; - goto loop; - } - mark_object (ptr->car); - obj = ptr->cdr; - cdr_count++; - if (cdr_count == mark_object_loop_halt) - abort (); - goto loop; + mark_object (ptr->car, depth + 1); + mark_object (ptr->cdr, depth + 1); + break; } case Lisp_Float: CHECK_ALLOCATED_AND_LIVE (live_float_p); + dump_object ("CONS", depth, obj, FLOAT_MARKED_P (XFLOAT (obj))); FLOAT_MARK (XFLOAT (obj)); break; case Lisp_Int: + dump_object_int (depth, obj); break; default: @@ -5124,8 +5234,9 @@ /* Mark the pointers in a buffer structure. */ static void -mark_buffer (buf) +mark_buffer (buf, depth) Lisp_Object buf; + unsigned depth; { register struct buffer *buffer = XBUFFER (buf); register Lisp_Object *ptr, tmp; @@ -5142,24 +5253,24 @@ if (buffer->overlays_before) { XSETMISC (tmp, buffer->overlays_before); - mark_object (tmp); + mark_object (tmp, depth + 1); } if (buffer->overlays_after) { XSETMISC (tmp, buffer->overlays_after); - mark_object (tmp); + mark_object (tmp, depth + 1); } for (ptr = &buffer->name; (char *)ptr < (char *)buffer + sizeof (struct buffer); ptr++) - mark_object (*ptr); + mark_object (*ptr, depth + 1); /* If this is an indirect buffer, mark its base buffer. */ if (buffer->base_buffer && !VECTOR_MARKED_P (buffer->base_buffer)) { XSETBUFFER (base_buffer, buffer->base_buffer); - mark_buffer (base_buffer); + mark_buffer (base_buffer, depth + 1); } } @@ -5599,6 +5710,71 @@ return Flist (8, consed); } + +static void *(*dump_hook_old_malloc) (size_t, const void *); +static void *(*dump_hook_old_realloc) (void *, size_t, const void *); +static void (*dump_hook_old_free) (void *, const void *); + +static void +dump_malloc_backtrace (void) +{ + void *trace[20]; + unsigned depth, i; + + depth = backtrace (trace, 20); + for (i = 1; i < depth; ++i) + { + fprintf (stderr, " %p", trace[i]); + } + fputs ("\n", stderr); +} + +static void * +dump_hook_malloc (size_t size, const void *caller) +{ + void *result; + __malloc_hook = dump_hook_old_malloc; + result = malloc(size); + fprintf (stderr, "@ A %p %u :", result, size); + dump_malloc_backtrace (); + __malloc_hook = &dump_hook_malloc; + return result; +} + +static void * +dump_hook_realloc (void *ptr, size_t size, const void *caller) +{ + void *result; + __realloc_hook = dump_hook_old_realloc; + result = realloc(ptr, size); + fprintf (stderr, "@ R %p %p %u :", ptr, result, size); + dump_malloc_backtrace (); + __realloc_hook = &dump_hook_realloc; + return result; +} + +static void +dump_hook_free (void *ptr, const void *caller) +{ + __free_hook = dump_hook_old_free; + free(ptr); + fprintf (stderr, "@ F %p :", ptr); + dump_malloc_backtrace (); + __free_hook = &dump_hook_free; +} + +DEFUN ("enable-malloc-tracing", Fenable_malloc_tracing, Senable_malloc_tracing, 0, 0, "", + doc: /* Enable tracing of memory allocations. */) + () +{ + dump_hook_old_malloc = __malloc_hook; + dump_hook_old_realloc = __realloc_hook; + dump_hook_old_free = __free_hook; + __malloc_hook = &dump_hook_malloc; + __realloc_hook = &dump_hook_realloc; + __free_hook = &dump_hook_free; +} + int suppress_checking; void die (msg, file, line) @@ -5792,8 +5968,10 @@ defsubr (&Smake_marker); defsubr (&Spurecopy); defsubr (&Sgarbage_collect); + defsubr (&Sgarbage_collect_dump); defsubr (&Smemory_limit); defsubr (&Smemory_use_counts); + defsubr (&Senable_malloc_tracing); #if GC_MARK_STACK == GC_USE_GCPROS_CHECK_ZOMBIES defsubr (&Sgc_status); --- emacs-upstream/src/bytecode.c 2004-09-27 21:33:01.000000000 +0200 +++ emacs-memory-dump/src/bytecode.c 2004-09-25 19:16:07.000000000 +0200 @@ -289,10 +289,10 @@ eassert (stack->top); for (obj = stack->bottom; obj <= stack->top; ++obj) - mark_object (*obj); + mark_object (*obj, 1); - mark_object (stack->byte_string); - mark_object (stack->constants); + mark_object (stack->byte_string, 1); + mark_object (stack->constants, 1); } } --- emacs-upstream/src/eval.c 2004-09-27 21:32:39.000000000 +0200 +++ emacs-memory-dump/src/eval.c 2004-09-25 20:23:47.000000000 +0200 @@ -3260,14 +3260,14 @@ for (backlist = backtrace_list; backlist; backlist = backlist->next) { - mark_object (*backlist->function); + mark_object (*backlist->function, 1); if (backlist->nargs == UNEVALLED || backlist->nargs == MANY) i = 0; else i = backlist->nargs - 1; for (; i >= 0; i--) - mark_object (backlist->args[i]); + mark_object (backlist->args[i], 1); } } --- emacs-upstream/src/fns.c 2004-09-27 21:32:39.000000000 +0200 +++ emacs-memory-dump/src/fns.c 2004-09-25 19:15:38.000000000 +0200 @@ -4804,13 +4804,13 @@ /* Make sure key and value survive. */ if (!key_known_to_survive_p) { - mark_object (HASH_KEY (h, i)); + mark_object (HASH_KEY (h, i), 1); marked = 1; } if (!value_known_to_survive_p) { - mark_object (HASH_VALUE (h, i)); + mark_object (HASH_VALUE (h, i), 1); marked = 1; } } --- emacs-upstream/src/keyboard.c 2004-09-27 21:32:39.000000000 +0200 +++ emacs-memory-dump/src/keyboard.c 2004-09-25 17:36:06.000000000 +0200 @@ -11425,7 +11425,8 @@ /* Mark the pointers in the kboard objects. Called by the Fgarbage_collector. */ void -mark_kboards () +mark_kboards (depth) + unsigned depth; { KBOARD *kb; Lisp_Object *p; @@ -11433,19 +11434,19 @@ { if (kb->kbd_macro_buffer) for (p = kb->kbd_macro_buffer; p < kb->kbd_macro_ptr; p++) - mark_object (*p); - mark_object (kb->Voverriding_terminal_local_map); - mark_object (kb->Vlast_command); - mark_object (kb->Vreal_last_command); - mark_object (kb->Vprefix_arg); - mark_object (kb->Vlast_prefix_arg); - mark_object (kb->kbd_queue); - mark_object (kb->defining_kbd_macro); - mark_object (kb->Vlast_kbd_macro); - mark_object (kb->Vsystem_key_alist); - mark_object (kb->system_key_syms); - mark_object (kb->Vdefault_minibuffer_frame); - mark_object (kb->echo_string); + mark_object (*p, depth + 1); + mark_object (kb->Voverriding_terminal_local_map, depth + 1); + mark_object (kb->Vlast_command, depth + 1); + mark_object (kb->Vreal_last_command, depth + 1); + mark_object (kb->Vprefix_arg, depth + 1); + mark_object (kb->Vlast_prefix_arg, depth + 1); + mark_object (kb->kbd_queue, depth + 1); + mark_object (kb->defining_kbd_macro, depth + 1); + mark_object (kb->Vlast_kbd_macro, depth + 1); + mark_object (kb->Vsystem_key_alist, depth + 1); + mark_object (kb->system_key_syms, depth + 1); + mark_object (kb->Vdefault_minibuffer_frame, depth + 1); + mark_object (kb->echo_string, depth + 1); } { struct input_event *event; @@ -11455,11 +11456,11 @@ event = kbd_buffer; if (event->kind != SELECTION_REQUEST_EVENT) { - mark_object (event->x); - mark_object (event->y); + mark_object (event->x, depth + 1); + mark_object (event->y, depth + 1); } - mark_object (event->frame_or_window); - mark_object (event->arg); + mark_object (event->frame_or_window, depth + 1); + mark_object (event->arg, depth + 1); } } } ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-28 21:00 ` Florian Weimer 2004-09-28 21:51 ` Florian Weimer @ 2004-09-29 16:39 ` Richard Stallman 2004-09-29 23:51 ` Kenichi Handa 2 siblings, 0 replies; 15+ messages in thread From: Richard Stallman @ 2004-09-29 16:39 UTC (permalink / raw) Cc: emacs-devel Thank you very much for tracking down a cause of a major memory leak. We have had a lot of trouble with that in Emacs, and other people have so far not had much success in debugging them. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-28 21:00 ` Florian Weimer 2004-09-28 21:51 ` Florian Weimer 2004-09-29 16:39 ` Richard Stallman @ 2004-09-29 23:51 ` Kenichi Handa 2 siblings, 0 replies; 15+ messages in thread From: Kenichi Handa @ 2004-09-29 23:51 UTC (permalink / raw) Cc: rms, emacs-devel In article <87y8iurtvw.fsf@deneb.enyo.de>, Florian Weimer <fw@deneb.enyo.de> writes: > It turns out that we are returning from code_convert_region() without > freeing the composition data under certain circumstances. This bug is > not new, it has been present since code_convert_region() started > allocating data using xmalloc() in late 1999. The recent > internationalization efforts just uncovered it. Thank you very much for finding this bug. I've just installed your fix. I also checked all calls of coding_save_composition and coding_allocate_composition_data, and installed similer fixes. --- Ken'ichi HANDA handa@m17n.org ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-28 15:20 ` Richard Stallman 2004-09-28 21:00 ` Florian Weimer @ 2004-09-28 21:40 ` Kim F. Storm 1 sibling, 0 replies; 15+ messages in thread From: Kim F. Storm @ 2004-09-28 21:40 UTC (permalink / raw) Cc: Florian Weimer, emacs-devel Richard Stallman <rms@gnu.org> writes: > It might be interesting to keep track of malloc'ing of large blocks of > memory, recording the last N large blocks allocated in a table. > Or perhaps only those that do not hold conses, miscs, small strings, etc. > This way you might be able to find some large blocks and then > examine them to see what data is in them. Other things that could be potential leaks are: - realized faces / fonts - cached images - buffer text -- Kim F. Storm <storm@cua.dk> http://www.cua.dk ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-21 17:38 Debugging memory leaks/stale references Florian Weimer 2004-09-21 19:49 ` Simon Josefsson @ 2004-09-21 19:57 ` Stefan Monnier 2004-09-21 21:01 ` Florian Weimer 1 sibling, 1 reply; 15+ messages in thread From: Stefan Monnier @ 2004-09-21 19:57 UTC (permalink / raw) Cc: emacs-devel > Is it possible to dump all reachable Lisp objects to a file, in the > format that is returned by princ? Not that I know. > I could run diff(1) on two versions of the file, and this might reveal > which objects are leaking (or held by stale references). That's an interesting idea. But one of the problems is that `princ' is not enough. Starting from the root set (i.e. the stack, and the global vars), princ would fail to print a lot of info because buffers are just printed #<buffer foo> without exposing its contents. > garbage-collect output alone is not very helpful. Indeed. One thing I was thinking about is a function `count-exclusive-children' which would take an object X as a parameter (could be the obarray, a buffer, a symbol, ...) and would return the number of objects that would die if X died. I think this can be implemented reasonably easily: 0 - change mark_object so as to increment a counter `objects_marked' each time it marks an object. 1 - set the mark bit on X (so as ot pretend it and its children were already marked). 2 - call the gc_mark routine (extracted from garbage_collect). 4 - reset objects_marked to 0. 3 - call `mark_object(X)'. 5 - read the value of objects_marked. 6 - do the gc_sweep. 7 - return the value read. Obviously it'd be a slow function (it does a full GC just to get a single integer number), but maybe it's good enough? You could start by calling (mapcar 'count-exclusive-children (buffer-list)) every once in a while to see how the numbers evolve. Or maybe (let ((res nil)) (mapatoms (lambda (s) (push (cons s count-exclusive-children s) res))) res) > (I'm going to test CVS Emacs soon, just to be sure that the behavior > is still the same. I think it's indeed the same with CVS, based on reports we got here not that long ago from Kai and others. > It's probably a Gnus bug, but you never know.) It might also be in Emacs or in an interaction between the two. I remember seeing strange things a couple years back where dead buffers where kept "alive" much longer than I thought (and they can hold on to a lot of data in their variables and overlays). Stefan ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Debugging memory leaks/stale references 2004-09-21 19:57 ` Stefan Monnier @ 2004-09-21 21:01 ` Florian Weimer 0 siblings, 0 replies; 15+ messages in thread From: Florian Weimer @ 2004-09-21 21:01 UTC (permalink / raw) Cc: emacs-devel * Stefan Monnier: >> Is it possible to dump all reachable Lisp objects to a file, in the >> format that is returned by princ? > > Not that I know. > >> I could run diff(1) on two versions of the file, and this might reveal >> which objects are leaking (or held by stale references). > > That's an interesting idea. > But one of the problems is that `princ' is not enough. Starting from the > root set (i.e. the stack, and the global vars), princ would fail to print > a lot of info because buffers are just printed #<buffer foo> without > exposing its contents. Hmm, I don't think so. You'd have to mimic a GC traversal, and GC does descend into buffer objects. > Indeed. One thing I was thinking about is a function > `count-exclusive-children' which would take an object X as a parameter > (could be the obarray, a buffer, a symbol, ...) and would return the number > of objects that would die if X died. I'm not sure about the usefulness of this function. If data structures are cyclic, it might not yield any actionable data at all. > I think it's indeed the same with CVS, based on reports we got here not that > long ago from Kai and others. Indeed. But at least my Emacs doesn't crash at the end of the day. >> It's probably a Gnus bug, but you never know.) > > It might also be in Emacs or in an interaction between the two. Or even with GNU libc's malloc(). The Emacs allocator disables allocation using mmap() in some cases, and the corresponding part of the libc allocator might have been subject to bit rot lately. ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2004-09-29 23:51 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2004-09-21 17:38 Debugging memory leaks/stale references Florian Weimer 2004-09-21 19:49 ` Simon Josefsson 2004-09-27 19:40 ` Florian Weimer 2004-09-27 19:52 ` Stefan Monnier 2004-09-27 20:36 ` Florian Weimer 2004-09-27 20:49 ` Stefan Monnier 2004-09-27 20:48 ` Simon Josefsson 2004-09-28 15:20 ` Richard Stallman 2004-09-28 21:00 ` Florian Weimer 2004-09-28 21:51 ` Florian Weimer 2004-09-29 16:39 ` Richard Stallman 2004-09-29 23:51 ` Kenichi Handa 2004-09-28 21:40 ` Kim F. Storm 2004-09-21 19:57 ` Stefan Monnier 2004-09-21 21:01 ` Florian Weimer
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).