From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Damien Mattei Newsgroups: gmane.lisp.guile.user Subject: Re: GC Warning: Repeated allocation of very large block Date: Mon, 19 Sep 2022 19:26:15 +0200 Message-ID: References: <874jx3612m.fsf@laura> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="24162"; mail-complaints-to="usenet@ciao.gmane.io" Cc: guile-user To: Olivier Dion Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Mon Sep 19 19:26:48 2022 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oaKXg-00067A-PG for guile-user@m.gmane-mx.org; Mon, 19 Sep 2022 19:26:48 +0200 Original-Received: from localhost ([::1]:54312 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oaKXf-0002Rg-Qm for guile-user@m.gmane-mx.org; Mon, 19 Sep 2022 13:26:47 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:52446) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oaKXP-0002Q9-N6 for guile-user@gnu.org; Mon, 19 Sep 2022 13:26:31 -0400 Original-Received: from mail-lf1-x12e.google.com ([2a00:1450:4864:20::12e]:44629) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oaKXM-0001GH-JT for guile-user@gnu.org; Mon, 19 Sep 2022 13:26:31 -0400 Original-Received: by mail-lf1-x12e.google.com with SMTP id i26so48089125lfp.11 for ; Mon, 19 Sep 2022 10:26:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=UFqDilpEC4MRFySLpdkiBf+u2rFAWE4NmQ+b9HK3lhs=; b=HkyI0vvjGs91Um7tPvgiPWKOtf7yeCtx5S3teVL4F53ZdNt5YoYQCnqOguUt7fTvh8 DmZaBoTgQConQvuvzyGW+btch54B3xM4SbRiVdRKB2Jp6EZjDTGu5h6wYd55yNBE1Ud6 L7lwUUCJC55CwvHCrzyTyim+t1+zf/kzJzcP5R9AGMejSU/ur3Matf4hVJ+GJ0+xugdl v+RrV+DNUF7C9om7W0vnApa5WO4uBBWKuT1LR1c7PukOEKg6kL0GfphIGQJYIPag+JgD 7RgCer1IXUhbGu0M1vSGSTTQPk/veEOe5o2PG9XVzbY8Ri6pPayuxWZbfLco9EvYSsU5 h91Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=UFqDilpEC4MRFySLpdkiBf+u2rFAWE4NmQ+b9HK3lhs=; b=vs1LMw1768CPIxovTqk/K7DrXvoIiXPQ33qQc1WiHCxMFF+o3pO8gw2sV32D5faoHY BJdC6v8gAYE9P85x0jk7lBN2OVSbmZFVSmic8C6AzUShR4omQusAcr6HK6MZVW8rQUEQ i38J5UkysBChB7TIblhYgsdE6BT4Tn47oM48maWyiNwNKZsLffp5pEscwq6DWToU4eIo V22kisr/Z6wwUMFpBDAwhslf/hRtHq3lwvSt3DEYKqNPCmF7EOGyg1FogVwAhIJ2gqgs VkdnSJi6qh/ZCfVfoi9mrQCh7WbtfH15xd/v+zs9onHFpxUdX0zolgsoPjb2541DjaxW DPvQ== X-Gm-Message-State: ACrzQf0hKKdmtCNNw+bHtpuwaKzXpOJ+w1GBiHMv4u6CgMmywM+/c30O IDjiDWS9EowUDvHHh5raxFC0KvRWfI9AkMPRxcA= X-Google-Smtp-Source: AMsMyM6s2yaAaNcm7Y0yITEewIgu3jGxtWFiqVJUU1oDuS0wsyfUPeGvylT1w3Vzffbus1Jj4gBoZA751xwSJYjJD6I= X-Received: by 2002:a05:6512:114b:b0:49d:4482:22b1 with SMTP id m11-20020a056512114b00b0049d448222b1mr6387971lfg.338.1663608386581; Mon, 19 Sep 2022 10:26:26 -0700 (PDT) In-Reply-To: <874jx3612m.fsf@laura> Received-SPF: pass client-ip=2a00:1450:4864:20::12e; envelope-from=damien.mattei@gmail.com; helo=mail-lf1-x12e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:18597 Archived-At: thank you, i think my algorithm is right but can be improved, i have to work on it,activate tracing to see where is the big expression list and if it is possible to correct that. There is a main loop in code and the warning only appear when data is bigger so i do not think of any memory leak (only have alittle doubt on an hastable which is global variable and reused: (when debug-mode (display-nl "Quine-Mc-Cluskey:")) {minterms-ht <- (make-hash-table)} ;; need to be cleared at each run (init-hash-table-with-set-and-value minterms-ht minterms #f) this : ;; the hash table for minterms, better to be a top-level definition,it's nightmare otherwise... (declare minterms-ht) {minterms-ht <- (make-hash-table)} ;; Scheme+ syntax expression create a new hash table with pointer minterms-ht this is done at each loop in the for each Cn and i should clear it before but there is no instruction to do that in the SRFI 69: https://srfi.schemers.org/srfi-69/srfi-69.html ??? I will also check the still running memory used by program. Damien On Mon, Sep 19, 2022 at 1:44 PM Olivier Dion wrote: > On Mon, 19 Sep 2022, Damien Mattei wrote: > > is this message appearing when a single scheme variable reach a given > > size? > > This message is from the bdwgc and not from Guile itself. From their > documentation: > --8<---------------cut here---------------start------------->8--- > The garbage collector generates warning messages of the form: > > > Repeated allocation of very large block ... > May lead to memory leak and poor performance > > > when it needs to allocate a block at a location that it knows to be > referenced > by a false pointer. These false pointers can be either permanent (e.g. > a static integer variable that never changes) or temporary. In the latter > case, the warning is largely spurious, and the block will eventually > be reclaimed normally. In the former case, the program will still run > correctly, but the block will never be reclaimed. Unless the block is > intended > to be permanent, the warning indicates a memory leak. > > 1. Ignore these warnings while you are using GC_DEBUG. Some of the > routines > mentioned below don't have debugging equivalents. (Alternatively, write > the > missing routines and send them to me.) > 2. Replace allocator calls that request large blocks with calls to > `GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You > may > want to set a breakpoint in `GC_default_warn_proc` to help you identify > such > calls. Make sure that a pointer to somewhere near the beginning of the > resulting block is maintained in a (preferably volatile) variable as long > as the block is needed. > 3. If the large blocks are allocated with realloc, we suggest instead > allocating them with something like the following. Note that the realloc > size increment should be fairly large (e.g. a factor of 3/2) for this to > exhibit reasonable performance. But we all know we should do that anyway. > > > void * big_realloc(void *p, size_t new_size) { > size_t old_size = GC_size(p); > void * result; > if (new_size <= 10000) return(GC_realloc(p, new_size)); > if (new_size <= old_size) return(p); > result = GC_malloc_ignore_off_page(new_size); > if (result == 0) return(0); > memcpy(result,p,old_size); > GC_free(p); > return(result); > } > > > 4. In the unlikely case that even relatively small object (<20 KB) > allocations are triggering these warnings, then your address space > contains > lots of "bogus pointers", i.e. values that appear to be pointers but > aren't. > Usually this can be solved by using `GC_malloc_atomic` or the routines > in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc. > Sometimes the problem can be solved with trivial changes of encoding > in certain values. It is possible, to identify the source of the bogus > pointers by building the collector with `-DPRINT_BLACK_LIST`, which will > cause it to print the "bogus pointers", along with their location. > 5. If you get only a fixed number of these warnings, you are probably > only > introducing a bounded leak by ignoring them. If the data structures being > allocated are intended to be permanent, then it is also safe to ignore > them. > The warnings can be turned off by calling `GC_set_warn_proc` with > a procedure that ignores these warnings (e.g. by doing absolutely > nothing). > --8<---------------cut here---------------end--------------->8--- > > It's my understanding that either A) there's a leak or B) its a false > positive. > > > i wanted to debug and trace because i know this algorithm expand a lot > > expressions (perheaps too much) sometimes (but it is an NP-problem and > > exponential, so perheaps stop the program is the only solution and run it > > on more little data) > > i do not think there could be a memory leak in a recursive scheme > > program, > > Gotta be careful with a conservative GC I guess. Your memory will get > reclaim at some point, but it's not guaranteed when since any value on > the C stack -- and global variables -- could reference your allocation > by accident. So if by accident there's a static constant value in C > that has the value of an allocation by the GC, it will never reclaim it. > Since it's the Scheme stack and I suppose it's tail call optimized, that > should not impact the GC. > > The most important thing would be to check the memory usage of the > program with a tool like `htop'. If there's leak, you will see that the > memory usage percentage keep increasing. > > -- > Olivier Dion > oldiob.dev >