From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Olivier Dion via General Guile related discussions Newsgroups: gmane.lisp.guile.user Subject: Re: GC Warning: Repeated allocation of very large block Date: Mon, 19 Sep 2022 07:44:17 -0400 Message-ID: <874jx3612m.fsf@laura> References: Reply-To: Olivier Dion Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="24192"; mail-complaints-to="usenet@ciao.gmane.io" To: Damien Mattei , guile-user Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Mon Sep 19 13:59:20 2022 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oaFQl-000672-PN for guile-user@m.gmane-mx.org; Mon, 19 Sep 2022 13:59:19 +0200 Original-Received: from localhost ([::1]:53072 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oaFQk-0007U2-Nc for guile-user@m.gmane-mx.org; Mon, 19 Sep 2022 07:59:18 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:41886) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oaFCP-00074t-Fv for guile-user@gnu.org; Mon, 19 Sep 2022 07:44:32 -0400 Original-Received: from smtp.polymtl.ca ([132.207.4.11]:39847) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oaFCM-0003ic-D3 for guile-user@gnu.org; Mon, 19 Sep 2022 07:44:28 -0400 Original-Received: from localhost (modemcable094.169-200-24.mc.videotron.ca [24.200.169.94]) by smtp.polymtl.ca (8.14.7/8.14.7) with ESMTP id 28JBiI4M008045; Mon, 19 Sep 2022 07:44:22 -0400 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp.polymtl.ca 28JBiI4M008045 In-Reply-To: X-Poly-FromMTA: (modemcable094.169-200-24.mc.videotron.ca [24.200.169.94]) at Mon, 19 Sep 2022 11:44:18 +0000 Received-SPF: pass client-ip=132.207.4.11; envelope-from=olivier.dion@polymtl.ca; helo=smtp.polymtl.ca X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:18596 Archived-At: On Mon, 19 Sep 2022, Damien Mattei wrote: > is this message appearing when a single scheme variable reach a given > size? This message is from the bdwgc and not from Guile itself. From their documentation: --8<---------------cut here---------------start------------->8--- The garbage collector generates warning messages of the form: Repeated allocation of very large block ... May lead to memory leak and poor performance when it needs to allocate a block at a location that it knows to be referenced by a false pointer. These false pointers can be either permanent (e.g. a static integer variable that never changes) or temporary. In the latter case, the warning is largely spurious, and the block will eventually be reclaimed normally. In the former case, the program will still run correctly, but the block will never be reclaimed. Unless the block is intended to be permanent, the warning indicates a memory leak. 1. Ignore these warnings while you are using GC_DEBUG. Some of the routines mentioned below don't have debugging equivalents. (Alternatively, write the missing routines and send them to me.) 2. Replace allocator calls that request large blocks with calls to `GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You may want to set a breakpoint in `GC_default_warn_proc` to help you identify such calls. Make sure that a pointer to somewhere near the beginning of the resulting block is maintained in a (preferably volatile) variable as long as the block is needed. 3. If the large blocks are allocated with realloc, we suggest instead allocating them with something like the following. Note that the realloc size increment should be fairly large (e.g. a factor of 3/2) for this to exhibit reasonable performance. But we all know we should do that anyway. void * big_realloc(void *p, size_t new_size) { size_t old_size = GC_size(p); void * result; if (new_size <= 10000) return(GC_realloc(p, new_size)); if (new_size <= old_size) return(p); result = GC_malloc_ignore_off_page(new_size); if (result == 0) return(0); memcpy(result,p,old_size); GC_free(p); return(result); } 4. In the unlikely case that even relatively small object (<20 KB) allocations are triggering these warnings, then your address space contains lots of "bogus pointers", i.e. values that appear to be pointers but aren't. Usually this can be solved by using `GC_malloc_atomic` or the routines in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc. Sometimes the problem can be solved with trivial changes of encoding in certain values. It is possible, to identify the source of the bogus pointers by building the collector with `-DPRINT_BLACK_LIST`, which will cause it to print the "bogus pointers", along with their location. 5. If you get only a fixed number of these warnings, you are probably only introducing a bounded leak by ignoring them. If the data structures being allocated are intended to be permanent, then it is also safe to ignore them. The warnings can be turned off by calling `GC_set_warn_proc` with a procedure that ignores these warnings (e.g. by doing absolutely nothing). --8<---------------cut here---------------end--------------->8--- It's my understanding that either A) there's a leak or B) its a false positive. > i wanted to debug and trace because i know this algorithm expand a lot > expressions (perheaps too much) sometimes (but it is an NP-problem and > exponential, so perheaps stop the program is the only solution and run it > on more little data) > i do not think there could be a memory leak in a recursive scheme > program, Gotta be careful with a conservative GC I guess. Your memory will get reclaim at some point, but it's not guaranteed when since any value on the C stack -- and global variables -- could reference your allocation by accident. So if by accident there's a static constant value in C that has the value of an allocation by the GC, it will never reclaim it. Since it's the Scheme stack and I suppose it's tail call optimized, that should not impact the GC. The most important thing would be to check the memory usage of the program with a tool like `htop'. If there's leak, you will see that the memory usage percentage keep increasing. -- Olivier Dion oldiob.dev