From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Andy Wingo Newsgroups: gmane.lisp.guile.devel Subject: Re: [PATCH] Configure GMP to use GC allocation functions, remove bignum finalizers Date: Tue, 29 Nov 2011 00:50:07 +0100 Message-ID: <87liqzajtc.fsf@pobox.com> References: <87tycaodlk.fsf@netris.org> <87k4d6edvr.fsf@gnu.org> <8739jue9wk.fsf@neil-laptop.ossau.uklinux.net> <87zkfh9x8r.fsf@pobox.com> <8762i5mf4x.fsf@gnu.org> <87r50ranub.fsf@pobox.com> <87zkff6dyr.fsf@gnu.org> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: dough.gmane.org 1322524226 11351 80.91.229.12 (28 Nov 2011 23:50:26 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Mon, 28 Nov 2011 23:50:26 +0000 (UTC) Cc: guile-devel@gnu.org, Neil Jerram To: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Tue Nov 29 00:50:21 2011 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1RVAxp-0000Vl-9h for guile-devel@m.gmane.org; Tue, 29 Nov 2011 00:50:21 +0100 Original-Received: from localhost ([::1]:52863 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVAxo-000587-2I for guile-devel@m.gmane.org; Mon, 28 Nov 2011 18:50:20 -0500 Original-Received: from eggs.gnu.org ([140.186.70.92]:57303) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVAxk-00057s-KX for guile-devel@gnu.org; Mon, 28 Nov 2011 18:50:18 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RVAxi-0007CM-7T for guile-devel@gnu.org; Mon, 28 Nov 2011 18:50:16 -0500 Original-Received: from a-pb-sasl-sd.pobox.com ([74.115.168.62]:35681 helo=sasl.smtp.pobox.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVAxi-0007C2-3x; Mon, 28 Nov 2011 18:50:14 -0500 Original-Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id 6BB237E5B; Mon, 28 Nov 2011 18:50:13 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=eDfj8q8S9hK3ks4ZOeiTmFb7Odk=; b=eM4GMq zlLd/qczW+Z3bAHmcYQVQ0Ow60PK45yDI9ffoP0Njz+WTWru/wEQw51NqxrR/668 My7Cs+AsiGSVRLZUVnzskXh8aSMOQ+OtF+oLzXl589zVNXoiM5K+shVg5jpyMELE CMgns+hiensco3kWFIYmE7fmWa4/tcSibz+qY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=sasl; b=O7PgCQoUF9D53ZhGYl+O6AN+jGXYNCKU LO1xbug/cv1JKfKTNoNjiqXBs+szFBltmgR/L28QmRrZQWt79ZJkAD2fdpa/XCGd 931/5hMIxlDpJgGZsCXDOO/NZzLXE5kGpKq8ylDYo6ufpF+aWflRKlUijdIiXFdx uRIqpdH+BIk= Original-Received: from a-pb-sasl-sd.pobox.com (unknown [127.0.0.1]) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id 642B07E5A; Mon, 28 Nov 2011 18:50:13 -0500 (EST) Original-Received: from badger (unknown [91.117.99.191]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTPSA id B05CB7E59; Mon, 28 Nov 2011 18:50:11 -0500 (EST) In-Reply-To: <87zkff6dyr.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Tue, 29 Nov 2011 00:10:04 +0100") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux) X-Pobox-Relay-ID: B60D05C4-1A1B-11E1-A43A-65B1DE995924-02397024!a-pb-sasl-sd.pobox.com X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-Received-From: 74.115.168.62 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:12940 Archived-At: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Tue 29 Nov 2011 00:10, ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Andy Wingo skribis: > >> On Sun 27 Nov 2011 22:25, ludo@gnu.org (Ludovic Court=C3=A8s) writes: >> >>> The problem is that this measurement doesn=E2=80=99t allow us to differ= entiate >>> between a growing heap with objects that may be freed as a result of >>> running the GC, and a growing heap just because the application needs >>> more malloc=E2=80=99d objects. >> >> This is true, but typically the heap stabilizes at some point. > > Ooh, re-reading your previous message, I now see what you mean, and it > makes sense to me. WDYT? --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=0001-increase-garbage-collection-rate-if-the-process-is-g.patch >From 5dc96678aba01297a127aa6f3781ecedec3b24f1 Mon Sep 17 00:00:00 2001 From: Andy Wingo Date: Tue, 29 Nov 2011 00:48:56 +0100 Subject: [PATCH] increase garbage collection rate if the process is growing * configure.ac: Check for GC_get_free_space_divisor. * libguile/gc.c (GC_get_free_space_divisor): Define an implementation, if needed. (accumulate_gc_timer): Fix indentation. (get_image_size): New terrible hack. Needs implementations on other platforms. (adjust_gc_frequency): Attempt to adjust the GC frequency based on process image growth. Needs more comments. (scm_init_gc): Add the adjust_gc_frequency to the after_gc_c_hook. --- configure.ac | 2 +- libguile/gc.c | 108 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 107 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 3a56cda..d63dd63 100644 --- a/configure.ac +++ b/configure.ac @@ -1259,7 +1259,7 @@ save_LIBS="$LIBS" LIBS="$BDW_GC_LIBS $LIBS" CFLAGS="$BDW_GC_CFLAGS $CFLAGS" -AC_CHECK_FUNCS([GC_do_blocking GC_call_with_gc_active GC_pthread_exit GC_pthread_cancel GC_allow_register_threads GC_pthread_sigmask GC_set_start_callback GC_get_heap_usage_safe]) +AC_CHECK_FUNCS([GC_do_blocking GC_call_with_gc_active GC_pthread_exit GC_pthread_cancel GC_allow_register_threads GC_pthread_sigmask GC_set_start_callback GC_get_heap_usage_safe GC_get_free_space_divisor]) # Though the `GC_do_blocking ()' symbol is present in GC 7.1, it is not # declared, and has a different type (returning void instead of diff --git a/libguile/gc.c b/libguile/gc.c index 4895b7d..281efde 100644 --- a/libguile/gc.c +++ b/libguile/gc.c @@ -27,6 +27,7 @@ #include #include #include +#include #ifdef __ia64__ #include @@ -194,6 +195,8 @@ SCM_DEFINE (scm_set_debug_cell_accesses_x, "set-debug-cell-accesses!", 1, 0, 0, +/* Compatibility. */ + #ifndef HAVE_GC_GET_HEAP_USAGE_SAFE static void GC_get_heap_usage_safe (GC_word *pheap_size, GC_word *pfree_bytes, @@ -208,6 +211,14 @@ GC_get_heap_usage_safe (GC_word *pheap_size, GC_word *pfree_bytes, } #endif +#ifndef HAVE_GC_GET_FREE_SPACE_DIVISOR +static GC_word +GC_get_free_space_divisor (void) +{ + return GC_free_space_divisor; +} +#endif + /* Hooks. */ scm_t_c_hook scm_before_gc_c_hook; @@ -230,6 +241,9 @@ unsigned long scm_gc_ports_collected = 0; static long gc_time_taken = 0; static long gc_start_time = 0; +static unsigned long free_space_divisor; +static unsigned long minimum_free_space_divisor; +static double target_free_space_divisor; static unsigned long protected_obj_count = 0; @@ -598,7 +612,10 @@ void scm_storage_prehistory () { GC_all_interior_pointers = 0; - GC_set_free_space_divisor (scm_getenv_int ("GC_FREE_SPACE_DIVISOR", 3)); + free_space_divisor = scm_getenv_int ("GC_FREE_SPACE_DIVISOR", 3); + minimum_free_space_divisor = free_space_divisor; + target_free_space_divisor = free_space_divisor; + GC_set_free_space_divisor (free_space_divisor); GC_INIT (); @@ -742,7 +759,8 @@ accumulate_gc_timer (void * hook_data SCM_UNUSED, void *data SCM_UNUSED) { if (gc_start_time) - { long now = scm_c_get_internal_run_time (); + { + long now = scm_c_get_internal_run_time (); gc_time_taken += now - gc_start_time; gc_start_time = 0; } @@ -750,6 +768,91 @@ accumulate_gc_timer (void * hook_data SCM_UNUSED, return NULL; } +static size_t +get_image_size (void) +{ + unsigned long size, resident, share; + size_t ret; + + FILE *fp = fopen ("/proc/self/statm", "r"); + + if (fp && fscanf (fp, "%lu %lu %lu", &size, &resident, &share) == 3) + ret = resident * 4096; + + if (fp) + fclose (fp); + + return ret; +} + +static void * +adjust_gc_frequency (void * hook_data SCM_UNUSED, + void *fn_data SCM_UNUSED, + void *data SCM_UNUSED) +{ + static size_t prev_image_size = 0; + static size_t prev_bytes_alloced = 0; + size_t image_size; + size_t bytes_alloced; + + image_size = get_image_size (); + bytes_alloced = GC_get_total_bytes (); + +#define HEURISTICS_DEBUG 0 + +#if HEURISTICS_DEBUG + fprintf (stderr, "prev image / alloced: %lu / %lu\n", prev_image_size, prev_bytes_alloced); + fprintf (stderr, " image / alloced: %lu / %lu\n", image_size, bytes_alloced); + fprintf (stderr, "divisor %lu / %f\n", free_space_divisor, target_free_space_divisor); +#endif + + if (prev_image_size && bytes_alloced != prev_bytes_alloced) + { + double growth_rate, new_target_free_space_divisor; + double damping_factor = 0.5; + double hysteresis = 0.1; + + growth_rate = ((double) image_size - prev_image_size) + / ((double)bytes_alloced - prev_bytes_alloced); + +#if HEURISTICS_DEBUG + fprintf (stderr, "growth rate %f\n", growth_rate); +#endif + + new_target_free_space_divisor = minimum_free_space_divisor; + + if (growth_rate > 0) + new_target_free_space_divisor *= 1.0 + growth_rate; + +#if HEURISTICS_DEBUG + fprintf (stderr, "new divisor %f\n", new_target_free_space_divisor); +#endif + + target_free_space_divisor = + (damping_factor * target_free_space_divisor + + (1.0 - damping_factor) * new_target_free_space_divisor); + +#if HEURISTICS_DEBUG + fprintf (stderr, "new target divisor %f\n", target_free_space_divisor); +#endif + + if (free_space_divisor + 0.5 + hysteresis < target_free_space_divisor + || free_space_divisor - 0.5 - hysteresis > target_free_space_divisor) + { + free_space_divisor = lround (target_free_space_divisor); +#if HEURISTICS_DEBUG + fprintf (stderr, "new divisor %lu\n", free_space_divisor); +#endif + GC_set_free_space_divisor (free_space_divisor); + } + } + + prev_image_size = image_size; + prev_bytes_alloced = bytes_alloced; + + return NULL; +} + @@ -847,6 +950,7 @@ scm_init_gc () scm_c_hook_add (&scm_before_gc_c_hook, queue_after_gc_hook, NULL, 0); scm_c_hook_add (&scm_before_gc_c_hook, start_gc_timer, NULL, 0); scm_c_hook_add (&scm_after_gc_c_hook, accumulate_gc_timer, NULL, 0); + scm_c_hook_add (&scm_after_gc_c_hook, adjust_gc_frequency, NULL, 0); #ifdef HAVE_GC_SET_START_CALLBACK GC_set_start_callback (run_before_gc_c_hook); -- 1.7.7.3 --=-=-= -- http://wingolog.org/ --=-=-=--