From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Damien Mattei Newsgroups: gmane.lisp.guile.user Subject: Re: GC Warning: Repeated allocation of very large block Date: Wed, 21 Sep 2022 11:48:05 +0200 Message-ID: References: <874jx3612m.fsf@laura> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="40019"; mail-complaints-to="usenet@ciao.gmane.io" Cc: guile-user To: Olivier Dion Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Wed Sep 21 11:53:43 2022 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oawQJ-000AFU-K6 for guile-user@m.gmane-mx.org; Wed, 21 Sep 2022 11:53:43 +0200 Original-Received: from localhost ([::1]:55210 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oawQI-00063S-NJ for guile-user@m.gmane-mx.org; Wed, 21 Sep 2022 05:53:42 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:58138) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oawLB-0000zM-Gf for guile-user@gnu.org; Wed, 21 Sep 2022 05:48:33 -0400 Original-Received: from mail-lf1-x131.google.com ([2a00:1450:4864:20::131]:34752) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oawL4-0003Tw-Gk for guile-user@gnu.org; Wed, 21 Sep 2022 05:48:25 -0400 Original-Received: by mail-lf1-x131.google.com with SMTP id j16so8297431lfg.1 for ; Wed, 21 Sep 2022 02:48:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=MENNbdZxOhk4N3JZzxl9Q/MOWoPP9azR0cyXJa/Xepw=; b=ZUcckbxrSFJfb1Mc5bEwPdE18z/JW+t+hqqXQRDR7iG3NRCPcwGwbrl7sW5NHISEjG LW/mF/1H5me73H2sPQeCMi8kW9Fz6Tf4ErGwsRdS5sRrGhSM1aAfYGreZQSyAQwXWIkP CUgkGnqFcx7XxaSHbK28eo84QXS6ddEiKkMZ1B/cIVunAyvUh+Gdjnpblq+u+CDb9ub8 QvsQlbwDrIrwxypMewBLSjyb3+ULLWlbQEfoFixTnPAg2rTyB/3jeNRwOczsqiVzlc5+ yVPiMDrOxZBXq7cPJBReDezLdErvRqFgjjMcJr7fjxMS1a0LwF9wwp7fhM5K/t/qkTsK hBtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=MENNbdZxOhk4N3JZzxl9Q/MOWoPP9azR0cyXJa/Xepw=; b=t2DIDXAsnuSbN53wDLma3kg+Yznu31yVE0JAAFCwIc7Kh3euB98gvQ7z+7mwzTU+o1 bHFRbahwgBx9BW4Zd7TMRhSJr+85/M+9kWDYt5sSULFij38048VleOXn+TVb1h8VglbI wuK+QZYmj+aw8lk8FaR1SEj/XgsR3/TMvuEgWp+ADrzSbGbS2zw8dRbC5nNDabXRPbaf 2AA0hyBJOLr3ouxkxVOIr6pieyl95w0SA84TcDMyPCRb4cLCCkbCR4d+SWxUZ1WUFsbo ozQ+KXQyKrnRKeBGijZCINNPSqYXo7tlGaBRc14dKDuss07sFzwM4AhmjOptbZGPW2h4 Eecg== X-Gm-Message-State: ACrzQf2sa8Zg9Nvi69RGlx+F5N8tj4YSstux6ip2jmMT1RA4IiKTjgSB 3VMunXroCTc4s/E8AIlP0dkp/tF1jeIX568jc/E= X-Google-Smtp-Source: AMsMyM4PHMYq4InBfnVm6SOx+p/BxmpRXS5ZQvtQfk1Keyfcx4m2+CZLsI9VoOY3zFi448HUOHItkd2LMwWpR49G2qA= X-Received: by 2002:a19:6b01:0:b0:499:b6fb:fcf2 with SMTP id d1-20020a196b01000000b00499b6fbfcf2mr8916839lfa.622.1663753696501; Wed, 21 Sep 2022 02:48:16 -0700 (PDT) In-Reply-To: Received-SPF: pass client-ip=2a00:1450:4864:20::131; envelope-from=damien.mattei@gmail.com; helo=mail-lf1-x131.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_SPF_HELO_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:18599 Archived-At: what is strange is how the Boehm garbage collector can have noticed that this repeated allocation of memory was a bug?, as it could have be intentional and normal.... Damien On Wed, Sep 21, 2022 at 11:42 AM Damien Mattei wrote: > i think the warning is not a false true because the program at reached in > 10 days almost 3Gb : > > PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM. > > > 332467 mattei 20 0 3262260 2,8g 3744 R 215,0 18,4 32277:54 > guile > > stopping it has released 3 Gb of memory, the problem can belong from the > hash table which is declared (not defined) with one identifier at top-level > but created in a function used many times in a loop so there was many > hash-table creates and the haskeys-values where not deleted when leaving > the function Quine-Mc-Cluskey that uses the hash-table because there is no > documentation and that in SRFI 69 i forget to do that, a simple function in > SRFI like this one could be useful: > > ;; delete all the elements of a hash table but not the hash table itself > (will be done by garbage collector) > > (define (hash-table-clear! ht) (map (lambda (key) (hash-table-delete! ht > key)) > (hash-table-keys ht))) > > i have modified and ran the program again, and will see if the message > from garbage collector happens again (in a week of computation...) i will > also check the whole program that seems to expand symbolic expressions a > lot when i have time. > > Regards, > Damien > > On Mon, Sep 19, 2022 at 7:26 PM Damien Mattei > wrote: > >> thank you, i think my algorithm is right but can be improved, i have to >> work on it,activate tracing to see where is the big expression list and if >> it is possible to correct that. There is a main loop in code and the >> warning only appear when data is bigger so i do not think of any memory >> leak (only have alittle doubt on an hastable which is global variable and >> reused: >> (when debug-mode (display-nl "Quine-Mc-Cluskey:")) >> {minterms-ht <- (make-hash-table)} ;; need to be cleared at each run >> (init-hash-table-with-set-and-value minterms-ht minterms #f) >> this : >> ;; the hash table for minterms, better to be a top-level definition,it's >> nightmare otherwise... >> (declare minterms-ht) >> {minterms-ht <- (make-hash-table)} ;; Scheme+ syntax >> expression create a new hash table with pointer minterms-ht >> this is done at each loop in the for each Cn >> and i should clear it before but there is no instruction to do that in >> the SRFI 69: >> https://srfi.schemers.org/srfi-69/srfi-69.html >> ??? >> I will also check the still running memory used by program. >> Damien >> >> On Mon, Sep 19, 2022 at 1:44 PM Olivier Dion >> wrote: >> >>> On Mon, 19 Sep 2022, Damien Mattei wrote: >>> > is this message appearing when a single scheme variable reach a given >>> > size? >>> >>> This message is from the bdwgc and not from Guile itself. From their >>> documentation: >>> --8<---------------cut here---------------start------------->8--- >>> The garbage collector generates warning messages of the form: >>> >>> >>> Repeated allocation of very large block ... >>> May lead to memory leak and poor performance >>> >>> >>> when it needs to allocate a block at a location that it knows to be >>> referenced >>> by a false pointer. These false pointers can be either permanent (e.g. >>> a static integer variable that never changes) or temporary. In the latter >>> case, the warning is largely spurious, and the block will eventually >>> be reclaimed normally. In the former case, the program will still run >>> correctly, but the block will never be reclaimed. Unless the block is >>> intended >>> to be permanent, the warning indicates a memory leak. >>> >>> 1. Ignore these warnings while you are using GC_DEBUG. Some of the >>> routines >>> mentioned below don't have debugging equivalents. (Alternatively, >>> write the >>> missing routines and send them to me.) >>> 2. Replace allocator calls that request large blocks with calls to >>> `GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You >>> may >>> want to set a breakpoint in `GC_default_warn_proc` to help you >>> identify such >>> calls. Make sure that a pointer to somewhere near the beginning of the >>> resulting block is maintained in a (preferably volatile) variable as >>> long >>> as the block is needed. >>> 3. If the large blocks are allocated with realloc, we suggest instead >>> allocating them with something like the following. Note that the >>> realloc >>> size increment should be fairly large (e.g. a factor of 3/2) for this >>> to >>> exhibit reasonable performance. But we all know we should do that >>> anyway. >>> >>> >>> void * big_realloc(void *p, size_t new_size) { >>> size_t old_size = GC_size(p); >>> void * result; >>> if (new_size <= 10000) return(GC_realloc(p, new_size)); >>> if (new_size <= old_size) return(p); >>> result = GC_malloc_ignore_off_page(new_size); >>> if (result == 0) return(0); >>> memcpy(result,p,old_size); >>> GC_free(p); >>> return(result); >>> } >>> >>> >>> 4. In the unlikely case that even relatively small object (<20 KB) >>> allocations are triggering these warnings, then your address space >>> contains >>> lots of "bogus pointers", i.e. values that appear to be pointers but >>> aren't. >>> Usually this can be solved by using `GC_malloc_atomic` or the routines >>> in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc. >>> Sometimes the problem can be solved with trivial changes of encoding >>> in certain values. It is possible, to identify the source of the bogus >>> pointers by building the collector with `-DPRINT_BLACK_LIST`, which >>> will >>> cause it to print the "bogus pointers", along with their location. >>> 5. If you get only a fixed number of these warnings, you are probably >>> only >>> introducing a bounded leak by ignoring them. If the data structures >>> being >>> allocated are intended to be permanent, then it is also safe to ignore >>> them. >>> The warnings can be turned off by calling `GC_set_warn_proc` with >>> a procedure that ignores these warnings (e.g. by doing absolutely >>> nothing). >>> --8<---------------cut here---------------end--------------->8--- >>> >>> It's my understanding that either A) there's a leak or B) its a false >>> positive. >>> >>> > i wanted to debug and trace because i know this algorithm expand a lot >>> > expressions (perheaps too much) sometimes (but it is an NP-problem and >>> > exponential, so perheaps stop the program is the only solution and run >>> it >>> > on more little data) >>> > i do not think there could be a memory leak in a recursive scheme >>> > program, >>> >>> Gotta be careful with a conservative GC I guess. Your memory will get >>> reclaim at some point, but it's not guaranteed when since any value on >>> the C stack -- and global variables -- could reference your allocation >>> by accident. So if by accident there's a static constant value in C >>> that has the value of an allocation by the GC, it will never reclaim it. >>> Since it's the Scheme stack and I suppose it's tail call optimized, that >>> should not impact the GC. >>> >>> The most important thing would be to check the memory usage of the >>> program with a tool like `htop'. If there's leak, you will see that the >>> memory usage percentage keep increasing. >>> >>> -- >>> Olivier Dion >>> oldiob.dev >>> >>