From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Linas Vepstas Newsgroups: gmane.lisp.guile.user Subject: Re: out-of-control GC Date: Thu, 14 Sep 2017 14:14:09 -0500 Message-ID: References: <87d16ve3oq.fsf@netris.org> Reply-To: linasvepstas@gmail.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Trace: blaine.gmane.org 1505416506 17971 195.159.176.226 (14 Sep 2017 19:15:06 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Thu, 14 Sep 2017 19:15:06 +0000 (UTC) Cc: Guile User To: Mark H Weaver Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Thu Sep 14 21:15:01 2017 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dsZbI-0004U4-7q for guile-user@m.gmane.org; Thu, 14 Sep 2017 21:15:00 +0200 Original-Received: from localhost ([::1]:49603 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dsZbP-0005fw-HT for guile-user@m.gmane.org; Thu, 14 Sep 2017 15:15:07 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:48397) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dsZar-0005e3-TC for guile-user@gnu.org; Thu, 14 Sep 2017 15:14:35 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dsZaq-0007Ih-5k for guile-user@gnu.org; Thu, 14 Sep 2017 15:14:33 -0400 Original-Received: from mail-lf0-x236.google.com ([2a00:1450:4010:c07::236]:52317) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dsZap-0007Ga-PE for guile-user@gnu.org; Thu, 14 Sep 2017 15:14:32 -0400 Original-Received: by mail-lf0-x236.google.com with SMTP id b127so299801lfe.9 for ; Thu, 14 Sep 2017 12:14:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc; bh=GHUkn225zP4KlgUzHt/GdwqVtwUrs8Dzbt848rZIk3U=; b=OViAh6zg1jkqyXOPRgLWOUuf6C5GVTGm5+MECas/bZoW6enouGa/e/02jUhzMHqzzt bD5CBOHVAE5UpwADOj7DK0QdOF5KtLrafKtMpcjxrVkpp2AWhRnTA/1/ZbmFF+kQqvLe 1UKDOJ3lmZtKpWjJWlAffebB5WCJcyTJvmVp8Aj3Rd5VMwIriLL0g6jMiq89I/EQ9cYM kzEk5QySclP2XHWgZMX+1vwkXGloClk6JioRLYqMWrCX+nXSELgJjX6eIXxeMDMWWGPs Mrtx96SHuIzeNtwbkhUfKO+qlnF97ru6t4KSG2humlNHs+3a+TnOz62rL9G8JoHT3EHS N6GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=GHUkn225zP4KlgUzHt/GdwqVtwUrs8Dzbt848rZIk3U=; b=QhwpzeVCCcdD1aioYZ0tqGMF+suGjIuoh6Tf4OOCKOubKD1WWML6EKXqC9TOB3cjFQ dhlbOAXRCHjwLXIBhuiK3NA6U4Z38y3hlvlVsFKtDKaakmZPiN9Na8PgMXQbrxHQyHDt otd8inGXdapaQ8h9bzAr6++Qi/LZYsCW5xjyP3RpZ1AnTZNFDUDnGehNeKYahTrOHwBQ 57arknY8JORmuGIbaAQNGM3o81+xDBIVzkzVNt6MI6jrJm888ak1q1ESnWW3h0u0Q7/S Lg5Cksuydk28xBemqGEazLiTAqGU0sgtchpM2NbKhQgwqgeK/UN3JIGjt1OWZtCYsxu7 CC4w== X-Gm-Message-State: AHPjjUgk8RcGYAbYU9wJWoUE0PAc94e2UZHRM4YTKDcFHZwF51FqxlJm m1OM12mzr7Wopn1Kk20JPZgcw2J+UmGTAPdOtR3a9A== X-Google-Smtp-Source: AOwi7QCZnkwN4DX8PoTvYAp5diaxJmkG6xuqKNPgHJnU9uCESIOAqbFM3c/R5Z15lvpB+rr4R53YajRbQYJu5W/68UU= X-Received: by 10.25.19.98 with SMTP id j95mr7980225lfi.77.1505416470448; Thu, 14 Sep 2017 12:14:30 -0700 (PDT) Original-Received: by 10.25.44.200 with HTTP; Thu, 14 Sep 2017 12:14:09 -0700 (PDT) In-Reply-To: <87d16ve3oq.fsf@netris.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4010:c07::236 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:14148 Archived-At: Hi Mark, Thanks... I'm not sure how to reply. So below I write "Yes, but...". My alternative was to ignore your email, but that seems rude. On Tue, Sep 12, 2017 at 9:24 PM, Mark H Weaver wrote: > Hi Linas, > > Linas Vepstas writes: > > > A few observations: > > * The (format #f ...) allocates a temporary string output port, and that > turns out to be quite a lot of allocation, especially in Guile 2.2. > Glancing at the relevant code, I would estimate that it's at least 2.5 > kilobytes spread over at least 10 heap blocks of various sizes. It > also involves registering a finalizer for the port, and adding to a > weak set, both of which are relatively expensive for the GC to deal > with. Using 'number->string' there would reduce memory allocation of > the loop above by an order of magnitude or more. > Yes but ... that code was meant to be an approximation of the "real" code. In the "real" code, there's maybe dozen-ish scheme calls, so the estimate of 2.5KB spread over 10 block might be in the right range. In the "real" code, there is also maybe 30KB of C++ mallocs, per iteration. Again, those mallocs should be "invisible" to bdwgc, but they might serve to fragment RAM badly. > > > * The fact that you're using 'set!' to mutate 'longer' forces a > heap-allocated variable object to be allocated for 'longer', whereas > otherwise it could be allocated on the stack. See section 9.3.4 > (Variables and the VM) in the Guile 2.2 manual for more on this, and > how 'set!' often makes things much less efficient. > Heh. I was using set! for this demo, only to clear out the long list. otherwise, you'd get a multi-billion-entry list, which clearly blows out the stack. > > * Since 'longer' is allocated within the loop, a new copy of that > (heap-allocated) variable object is created for each iteration. > Hmm. That's interesting, Is that really right? The thing was meant to be a tail recursive loop, designed to run for some arbitarily long time. Clearly a heap allocation in a loop would be a disaster. In my "real" code, I don't think I am using set! in this way (but I will now have to pay closer attention). Its not clear that this isn't some kind of guile design flaw -- I can't imagine a technical reason for why a set! to a local variable would force it to go on the heap. > Since you're running this loop about 1.2 billion times, and each > iteration is allocating at least 2.5k bytes, that comes out to around 3 > terabytes of allocation during this loop. All of that must be reclaimed > by the garbage collector. > Yeah, the 1.2 billion was to get it to run some 12+ hours, so that I could verify if it slowly degrades over time (it didn't - it was stable after the first few minutes). My loop was running 25 million times, and it took some 12 or 20 hours to finish. And it did seem to slowly degrade, but I'm not quite sure. > Also, since the port objects have finalizers, I don't think that any of my code is using ports in my loop, or that there's anything else in the loop that needs finalization. I need to create some better instrumentation. But at 1-3 days per run, debugging gets tedious very quickly very fast. One is not motivated to revisit the problem, once you finally come out the other end. Until, of course, next time. --linas -- *"The problem is not that artificial intelligence will get too smart and take over the world," computer scientist Pedro Domingos writes, "the problem is that it's too stupid and already has." *