From mboxrd@z Thu Jan 1 00:00:00 1970 Path: main.gmane.org!not-for-mail From: "Michael A. Wells" Newsgroups: gmane.lisp.guile.devel,gmane.lisp.guile.bugs Subject: substantial performance loss when running long-lived or computationally intensive programs Date: Mon, 05 May 2003 22:36:26 -0500 (CDT) Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Message-ID: <20030505.223626.84190828.wells@email.mot.com> Reply-To: michaelawells@motorola.com NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Trace: main.gmane.org 1052192149 18395 80.91.224.249 (6 May 2003 03:35:49 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Tue, 6 May 2003 03:35:49 +0000 (UTC) Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Tue May 06 05:35:47 2003 Return-path: Original-Received: from monty-python.gnu.org ([199.232.76.173]) by main.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 19CtF5-0004mX-00 for ; Tue, 06 May 2003 05:35:47 +0200 Original-Received: from localhost ([127.0.0.1] helo=monty-python.gnu.org) by monty-python.gnu.org with esmtp (Exim 4.10.13) id 19CtG5-0006ZX-05 for guile-devel@m.gmane.org; Mon, 05 May 2003 23:36:49 -0400 Original-Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.10.13) id 19CtFq-0006Rd-00 for guile-devel@gnu.org; Mon, 05 May 2003 23:36:34 -0400 Original-Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.10.13) id 19CtFo-0006NF-00 for guile-devel@gnu.org; Mon, 05 May 2003 23:36:33 -0400 Original-Received: from motgate4.mot.com ([144.189.100.102]) by monty-python.gnu.org with esmtp (Exim 4.10.13) id 19CtFo-0006JU-00; Mon, 05 May 2003 23:36:32 -0400 Original-Received: from az33exr04.mot.com (az33exr04.mot.com [10.64.251.234]) by motgate4.mot.com (Motorola/Motgate4) with ESMTP id h463aU9J012162; Mon, 5 May 2003 20:36:30 -0700 (MST) Original-Received: from magritte.stc.corp.mot.com (magritte.stc.corp.mot.com [199.15.71.39])h463aR5i028322; Mon, 5 May 2003 22:36:28 -0500 Original-Received: from localhost (localhost [127.0.0.1])h463aRo08610; Mon, 5 May 2003 22:36:27 -0500 (CDT) Original-To: bug-guile@gnu.org, guile-devel@gnu.org X-Mailer: Mew version 3.0.54 on Emacs 21.1 / Mule 5.0 (SAKAKI) X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1b5 Precedence: list List-Id: Developers list for Guile, the GNU extensibility library List-Help: List-Post: List-Subscribe: , List-Archive: List-Unsubscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: main.gmane.org gmane.lisp.guile.devel:2282 gmane.lisp.guile.bugs:784 X-Report-Spam: http://spam.gmane.org/gmane.lisp.guile.devel:2282 The Guile interpreter comes to a near halt after making a sufficiently large number of calls to guile built-ins which do not appropriately maintain the 'scm_mallocated' variable (declared in "libguile/gc.c"). Although this problem is present in Guile-1.6.4, the problem dates back to at least Guile 1.4. There appears to be a widespread misunderstanding as to how 'scm_mallocated' should be maintained. [Several other people have discussed this issue: http://mail.gnu.org/archive/html/bug-guile/2001-05/msg00024.html http://www.glug.org/snap/workbook/gc/memory.text ] There is no comment in "libguile/gc.c" which describes what 'scm_mallocated' should contain, but by looking through the code, it appears that 'scm_mallocated' should correspond to the total number of bytes obtained through malloc currently held by the interpreter. Each time a block of memory of size S is allocated, 'scm_mallocated' should be incremented by S. Each time a block of memory of size S is garbaged collected and/or freed, 'scm_mallocated' should be decremented by S. 'scm_mallocated', in combination with 'scm_mtrigger' is used to determine when the certain calls to the garbage collector (scm_igc) are triggered. When 'scm_mallocated' is greater than 'scm_mtrigger', garbage collection is triggered. After garbage collection, if the yield is too small, 'scm_mtrigger' is increased, relative to 'scm_mtrigger'. (See body of 'check_mtrigger' function.) [Garbage collections are triggered by functions other than 'scm_mtrigger', which turns out to be a good thing. If 'check_mtrigger' were the only caller of 'scm_igc', the size of the guile image could grow very rapidly.] Unfortunately, in some cases 'scm_mallocated' is incremented when the memory is allocated, but scm_mallocated' is not decremented when memory is freed. As 'scm_mallocated' approaches 2^32, the value of 'scm_mtrigger' may be set to a value _intended_ to be higher than 'scm_mallocated', but ends up wrapping around to a value lower than 'scm_mallocated'. It is also possible that 'scm_mallocated' will wrap around as well. Once a wraparound occurs, the interpreter comes to a near halt. I've identified a _partial_ list of code which increments but does not decrement 'scm_mallocated': libguile/filesys.c: getcwd, readline libguile/fports.c: fport_close libguile/posix.c: gethostname libguile/ports.c: scm_remove_from_port_table libguile/regex-posix.c: scm_regexp_exec These functions allocate memory with 'scm_must_malloc' and 'scm_must_realloc', then free memory with a call 'scm_must_free'. The call to 'scm_must_free' is not made by the garbage collector. While 'scm_must_malloc' and 'scm_must_realloc' increment 'scm_mallocated', 'scm_must_free' does not decrement 'scm_mallocated'. I have attached some scheme code which demonstrates how certain using certain guile built-ins increment but do not decrement 'scm_mallocated'. Thanks, Michael Wells ;---snip here (use-modules (ice-9 format)) (define (tester how-many-times thunk) ; force garbage collection (gc) (gc) (gc) ; get initial value of 'scm_mallocated' ; (available using 'bytes_malloced key from call to (gc-stats) (let ((initial-bytes-malloced (cdr (assoc 'bytes-malloced (gc-stats)))) (starting-seconds (tms:clock (times)))) ;(current-time))) ; run 'thunk' 'how-many-times' (do ((x 0 (+ x 1))) ((= x how-many-times) #t) (cond ((= (modulo x 1000) 0) (let ((now (tms:clock (times)))) (display (format " ~A : ~A clock units~%" x (- now starting-seconds))) (set! starting-seconds now)))) (thunk)) ; force garbage collection (gc) (gc) (gc) (let ((final-bytes-malloced (cdr (assoc 'bytes-malloced (gc-stats))))) (display (format " final-bytes-malloced: ~A~%" final-bytes-malloced)) (display (format "initial-bytes-malloced: ~A~%" initial-bytes-malloced)) (display (format " final - initial: ~A~%" (- final-bytes-malloced initial-bytes-malloced)))))) ; libguile/posix.c -- "gethostname" (tester 10000 (lambda () (gethostname))) ; ; Performance suffers when 'scm_mallocated' is very large. ; Note how quickly each 100 calls to '(gethostname)' ; run at first; compare with how long each 100 calls takes ; near end of run. ; (For me, performance grinds to a halt after the ; first 14779000 or so iterations of (gethostname). ;(tester 20000000 (lambda () (gethostname))) ; fports.c -- "fport_close" ; ports.c -- "scm_remove_from_port_table" (tester 1000 (lambda () (with-output-to-string (lambda () (display "1234567890") (display "1234567890"))))) ; libguile/regex-posix.c -- "scm_regexp_exec" (define *token* (make-regexp "^[A-Za-z0-9_]*" )) (tester 1000 (lambda () (regexp-exec *token* "foo"))) ; libguile/filesys.c "getcwd" (tester 1000 (lambda () (getcwd))) _______________________________________________ Guile-devel mailing list Guile-devel@gnu.org http://mail.gnu.org/mailman/listinfo/guile-devel