From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Damien Mattei Newsgroups: gmane.lisp.guile.user Subject: Re: GC Warning: Repeated allocation of very large block Date: Wed, 21 Sep 2022 11:42:29 +0200 Message-ID: References: <874jx3612m.fsf@laura> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="25407"; mail-complaints-to="usenet@ciao.gmane.io" Cc: guile-user To: Olivier Dion Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Wed Sep 21 11:43:03 2022 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oawFz-0006Nj-3l for guile-user@m.gmane-mx.org; Wed, 21 Sep 2022 11:43:03 +0200 Original-Received: from localhost ([::1]:35990 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oawFy-0006gW-2i for guile-user@m.gmane-mx.org; Wed, 21 Sep 2022 05:43:02 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:34438) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oawFi-0006eZ-AZ for guile-user@gnu.org; Wed, 21 Sep 2022 05:42:46 -0400 Original-Received: from mail-lf1-x136.google.com ([2a00:1450:4864:20::136]:34685) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oawFf-0002bH-KL for guile-user@gnu.org; Wed, 21 Sep 2022 05:42:46 -0400 Original-Received: by mail-lf1-x136.google.com with SMTP id j16so8274690lfg.1 for ; Wed, 21 Sep 2022 02:42:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=gWPdJI2XyPdBxIK+ioMo1WWRzUSc8KNR7K23wtUIuPs=; b=U1v73LYKct1v8lhP6Rcrw9CRF7TSD7wUyHj1MNZfX1Qd0Jcd4lT1jXxcNXBpB5QbuM ZINDxIQ8fLvM7BdPoKzvKvZ29r0S/dbH+nPRoswWIscAaFMXN2IVdUtgUWgv2n7+uDzR HH3jNCJ2QOQOJimg27jX/XwcRhOXRvcAYLRtCck+/gYJA3anPWhMIrFUXXycOxnMkIxE oIhPkrJ8yte6TyPAzLnPeChcbM8z7FLksn8JTX2RBIRZ1raLFyMce8gnI8itpz3D+OZ6 kcqoFCINIsy+B0YlTI3nf9jTmoP2GV66zBpe/PTPrcO/4H0148sN+AfhqCPHIwPDWHhE WI4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=gWPdJI2XyPdBxIK+ioMo1WWRzUSc8KNR7K23wtUIuPs=; b=sVhxG/Pce4t9kzLyEalqyjnunQniMViBeCWcuvo6yJOrv82P1zXzMEG+EZMGM33Ej0 ZwB4x7k6Dy9QGxOeOWKshhT6qfgUsFa1zskQx4B5CKI+huLiKNVtIT2ikT7ycfUF0+tj DGrRXafx76aEN+AZqjHKktKp2nb7YOewfzEmp6PHQQJmnEpK+Z9ncOSxrwJN5EB6uRkK zLaDN+Q9OL6NrhTY/e3fTOTFg2xeF6yrtgf00Qkue3KDWkng86CDgAvvCiioej35uXoT 8MllM/Dql7jmcW5Eip87z82NXlqKx199VGcxQ8P3Gmx2nC9z3AWnOolRrB9JST3ttY/7 CgYA== X-Gm-Message-State: ACrzQf17Jjdxkse5HD9oSr0kTuMqyxErs9Pdup+9ZCJRkg9OZfYlzANT XldfDcnxk9+HUpWmXwmCreSpTOKZ3+tlDyPU4rap1kSywNA= X-Google-Smtp-Source: AMsMyM5iEpHLA0HwtrZM3zlQVu7pJ7fNOZPsox3rE+qJbGOUdzA6Z1ET5QL7PhI2YUc2IfqNtSl2ScvYcBkt4SkgRLE= X-Received: by 2002:a19:6b01:0:b0:499:b6fb:fcf2 with SMTP id d1-20020a196b01000000b00499b6fbfcf2mr8908601lfa.622.1663753360450; Wed, 21 Sep 2022 02:42:40 -0700 (PDT) In-Reply-To: Received-SPF: pass client-ip=2a00:1450:4864:20::136; envelope-from=damien.mattei@gmail.com; helo=mail-lf1-x136.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:18598 Archived-At: i think the warning is not a false true because the program at reached in 10 days almost 3Gb : PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM. 332467 mattei 20 0 3262260 2,8g 3744 R 215,0 18,4 32277:54 guile stopping it has released 3 Gb of memory, the problem can belong from the hash table which is declared (not defined) with one identifier at top-level but created in a function used many times in a loop so there was many hash-table creates and the haskeys-values where not deleted when leaving the function Quine-Mc-Cluskey that uses the hash-table because there is no documentation and that in SRFI 69 i forget to do that, a simple function in SRFI like this one could be useful: ;; delete all the elements of a hash table but not the hash table itself (will be done by garbage collector) (define (hash-table-clear! ht) (map (lambda (key) (hash-table-delete! ht key)) (hash-table-keys ht))) i have modified and ran the program again, and will see if the message from garbage collector happens again (in a week of computation...) i will also check the whole program that seems to expand symbolic expressions a lot when i have time. Regards, Damien On Mon, Sep 19, 2022 at 7:26 PM Damien Mattei wrote: > thank you, i think my algorithm is right but can be improved, i have to > work on it,activate tracing to see where is the big expression list and if > it is possible to correct that. There is a main loop in code and the > warning only appear when data is bigger so i do not think of any memory > leak (only have alittle doubt on an hastable which is global variable and > reused: > (when debug-mode (display-nl "Quine-Mc-Cluskey:")) > {minterms-ht <- (make-hash-table)} ;; need to be cleared at each run > (init-hash-table-with-set-and-value minterms-ht minterms #f) > this : > ;; the hash table for minterms, better to be a top-level definition,it's > nightmare otherwise... > (declare minterms-ht) > {minterms-ht <- (make-hash-table)} ;; Scheme+ syntax > expression create a new hash table with pointer minterms-ht > this is done at each loop in the for each Cn > and i should clear it before but there is no instruction to do that in the > SRFI 69: > https://srfi.schemers.org/srfi-69/srfi-69.html > ??? > I will also check the still running memory used by program. > Damien > > On Mon, Sep 19, 2022 at 1:44 PM Olivier Dion > wrote: > >> On Mon, 19 Sep 2022, Damien Mattei wrote: >> > is this message appearing when a single scheme variable reach a given >> > size? >> >> This message is from the bdwgc and not from Guile itself. From their >> documentation: >> --8<---------------cut here---------------start------------->8--- >> The garbage collector generates warning messages of the form: >> >> >> Repeated allocation of very large block ... >> May lead to memory leak and poor performance >> >> >> when it needs to allocate a block at a location that it knows to be >> referenced >> by a false pointer. These false pointers can be either permanent (e.g. >> a static integer variable that never changes) or temporary. In the latter >> case, the warning is largely spurious, and the block will eventually >> be reclaimed normally. In the former case, the program will still run >> correctly, but the block will never be reclaimed. Unless the block is >> intended >> to be permanent, the warning indicates a memory leak. >> >> 1. Ignore these warnings while you are using GC_DEBUG. Some of the >> routines >> mentioned below don't have debugging equivalents. (Alternatively, write >> the >> missing routines and send them to me.) >> 2. Replace allocator calls that request large blocks with calls to >> `GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You >> may >> want to set a breakpoint in `GC_default_warn_proc` to help you identify >> such >> calls. Make sure that a pointer to somewhere near the beginning of the >> resulting block is maintained in a (preferably volatile) variable as >> long >> as the block is needed. >> 3. If the large blocks are allocated with realloc, we suggest instead >> allocating them with something like the following. Note that the realloc >> size increment should be fairly large (e.g. a factor of 3/2) for this to >> exhibit reasonable performance. But we all know we should do that >> anyway. >> >> >> void * big_realloc(void *p, size_t new_size) { >> size_t old_size = GC_size(p); >> void * result; >> if (new_size <= 10000) return(GC_realloc(p, new_size)); >> if (new_size <= old_size) return(p); >> result = GC_malloc_ignore_off_page(new_size); >> if (result == 0) return(0); >> memcpy(result,p,old_size); >> GC_free(p); >> return(result); >> } >> >> >> 4. In the unlikely case that even relatively small object (<20 KB) >> allocations are triggering these warnings, then your address space >> contains >> lots of "bogus pointers", i.e. values that appear to be pointers but >> aren't. >> Usually this can be solved by using `GC_malloc_atomic` or the routines >> in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc. >> Sometimes the problem can be solved with trivial changes of encoding >> in certain values. It is possible, to identify the source of the bogus >> pointers by building the collector with `-DPRINT_BLACK_LIST`, which will >> cause it to print the "bogus pointers", along with their location. >> 5. If you get only a fixed number of these warnings, you are probably >> only >> introducing a bounded leak by ignoring them. If the data structures >> being >> allocated are intended to be permanent, then it is also safe to ignore >> them. >> The warnings can be turned off by calling `GC_set_warn_proc` with >> a procedure that ignores these warnings (e.g. by doing absolutely >> nothing). >> --8<---------------cut here---------------end--------------->8--- >> >> It's my understanding that either A) there's a leak or B) its a false >> positive. >> >> > i wanted to debug and trace because i know this algorithm expand a lot >> > expressions (perheaps too much) sometimes (but it is an NP-problem and >> > exponential, so perheaps stop the program is the only solution and run >> it >> > on more little data) >> > i do not think there could be a memory leak in a recursive scheme >> > program, >> >> Gotta be careful with a conservative GC I guess. Your memory will get >> reclaim at some point, but it's not guaranteed when since any value on >> the C stack -- and global variables -- could reference your allocation >> by accident. So if by accident there's a static constant value in C >> that has the value of an allocation by the GC, it will never reclaim it. >> Since it's the Scheme stack and I suppose it's tail call optimized, that >> should not impact the GC. >> >> The most important thing would be to check the memory usage of the >> program with a tool like `htop'. If there's leak, you will see that the >> memory usage percentage keep increasing. >> >> -- >> Olivier Dion >> oldiob.dev >> >