From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Arne Babenhauserheide Newsgroups: gmane.lisp.guile.user Subject: Re: out-of-control GC Date: Mon, 11 Sep 2017 00:17:08 +0200 Message-ID: <87poay89ly.fsf@web.de> References: <87zia28phj.fsf@web.de> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Trace: blaine.gmane.org 1505321679 10916 195.159.176.226 (13 Sep 2017 16:54:39 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 13 Sep 2017 16:54:39 +0000 (UTC) Cc: "guile-user@gnu.org" To: linasvepstas@gmail.com Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Mon Sep 11 00:37:01 2017 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1drAqM-0008RD-Qg for guile-user@m.gmane.org; Mon, 11 Sep 2017 00:36:47 +0200 Original-Received: from localhost ([::1]:54490 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1drAqN-0000IV-Q0 for guile-user@m.gmane.org; Sun, 10 Sep 2017 18:36:47 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:40027) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1drApv-0000IA-8Z for guile-user@gnu.org; Sun, 10 Sep 2017 18:36:20 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1drAps-0006DF-2u for guile-user@gnu.org; Sun, 10 Sep 2017 18:36:19 -0400 Original-Received: from mout.web.de ([212.227.15.14]:59518) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1drApr-0006Cr-Pq for guile-user@gnu.org; Sun, 10 Sep 2017 18:36:16 -0400 Original-Received: from fluss ([85.212.77.149]) by smtp.web.de (mrweb002 [213.165.67.108]) with ESMTPSA (Nemesis) id 0LtFAh-1dT8bv47j9-012mnE; Mon, 11 Sep 2017 00:36:13 +0200 In-reply-to: X-Provags-ID: V03:K0:bdOOnU6AInijF95SLGK5E6TbBdCyf6/DrDhc/okdqULbtnUlW8P aw9bEUIv/l/ghcSZhCMb8DK6t22RHBKkyHfi1URiA0Nk/T2BQYrd5K2IIzw6awB0cOUbCFF WJUTrfYSUUaXu12GZUuBzLuh7qYNIgqc1HvNaif6OoJex+FXc9KsEheZ3TMbdDHfpw/2PT1 /9YhxSfFYOFQOVYVLLstA== X-UI-Out-Filterresults: notjunk:1;V01:K0:51HomPn215E=:zUooeTgUFchPtcGY9XnBxh Lqsv+vCOVHXVt0+v27DCgrM3uurFf/6IZFkWglZfn49Za23wCK5KjYcNcaQm6ZUjrEBUX6yZH ZXJeztL1/GxeM6G5IVHOFwUp+JhW6QnzJ0nmfBLrKjkZlpNydB2vl6cxziqE3YqkZN42r/PPL 6/JdHuqsq3hZ8d2idkJOLOD/fz/0n0gb9+aHZCIuu1lVGloZLR9pnWd50MCXX0La7jDeJh29o d6avMqCm1OBLC9ZN5msn0IXSBL1syn+GcQlgY6Bbi/UIkp13wYZ4NM1rFGTAolVusRVNuTjcY jDotmfVYx89tHFkPFVvW3NsXb3mxqjbIbM5XMLyeNyHuWn0Qg/p1jESwRxy8x7UCSzGKrlNOP bt9myfhA1pr6slbbI2gDl2kZCAvRFdzW0HUrMpgP/etXR/+xzL76S1Ws2BVKBBLTe9kW14U7W NNLt4DhqJwszcHw7rFc6U4fXJHjSMXCKSTMnL1uV12YgzSS0QuaWqek/8R1Yt/HiV2jPFnrBC n4nNF14r4Nc4xURBMM21+oanUqQL6VaaFFA78p/TPh6GrOQh6Vl1NVYx5rvgu2iJ0kSCMV1qB JeLAfcfBpM/0zM1VnfvNu0slceVtm9aRqaNLb2+SEe9SwPzzBcjqBv2NieCPNAI+CsjlfE+jX 87/89D+xS4tMo2jn8dKgVLe2gCajGtZVYpH4EhwqQfc1iApAKuM9tiOn1CB9JUPiu2FPdaNg5 gADBG8lzc4uI1YZDwlNLLuaw+x/LRCa/LPLVNqbaLSSgo+J4MGCPr7x06N5yGxe6BDHAkFZl X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 212.227.15.14 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:14132 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Linas, Linas Vepstas writes: >> If I replace >> >> (cons (format #f "~A" cnt) lis) >> >> with >> >> (cons cnt lis) >> >> GC is down to ~20% (from around 130%). >> > > That demo tried to capture the spirit of a more complex system. That=E2=80=99s great, by the way! I did that experiment, because the string formatting seems to create lots of values the GC has to collect. If this triggers an investigation of all memory, any reduction of disposable value creation should help. Just try putting a (gc-disable) just before you run make-bgl and see the memory usage shoot through the roof almost instantly. For the real-code: throw in a=20 > The cons > that I have is actually (cons (wrapper-for-some-c++-code foobar) lis) or > maybe (cons (scheme-that-calls-other-scheme-that-calls-c++ foo) lis) ... > but its not just cons -- it can also be for-each, map, or fold. > > The actual printouts, in my instrumented code look like this: > > guile-en-r> (avg-gc-cpu-time) > Elapsed: 4461. secs. Rate: 4.236 gc/min %cpu-GC: 280.8% %cpu-use: 284.0% This indeed looks pretty painful. > Anyway, this demo gives a scenario where gc seems to run too often. I ha= ve > other scenarios where it does not run often enough -- I have one where the > working set is about 200MBytes, but guile happily grows to 10GB or 20GB R= AM > over a few hours. My solution for that other case is to manually GC > whenever gc-stats gets above 1GB, because I know a priori my working set = is > smaller than that. > > The overall message seems to be that gc either runs too often, or not oft= en > enough. Some sort of better planning is needed. A short term solution could be to disable automatic triggering of GC in cases where it fires too often (compared to the total memory consumption). > I'm motivated to explore this, as it actually impacts my work. That=E2=80=99s great! And it=E2=80=99s awesome that you=E2=80=99re pushing = Guile that far. This is what it needs to become a reliable real-world battered workhorse for many more people. A start might be to look at the problem description: GC is triggered too often when your C++ datastructures are so huge that the growth in Guile-managed memory is irrelevant, and it is triggered too seldomly when you only manage a small amount of data. So maybe GC could benefit from a kind of hint from C++ about the size of the datastructures which allows tuning the GC frequency to the total amount of memory. =2D-=20 Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAlm1vlsACgkQE++NRSQD w+uOABAA00PMjPuvW+/gLmaJaNbhVyxv8hY5Ekj1klbZkILNB8xmPuDsjV0vjhsb OCKjslj5E5mI6HrZH097Qb5d8GfhNrk6rhdZR088red8wk8WPHtlDmY2x8q06eLz LUE87xJXmVYQIO3JfbM954LsaGk4wmrg9ppAL/yWknYYDCr31VOW0w2I5sxmDajU gpdWYEyiQoxDCZn28CVBn2T5uAueTA8byn8qCxydUtv1XHRYSGKbhleH7K1NdkKW YNviaWN1Vpc8wVmEEAgA//cwqr9we0eh4XfUI4oGbGLgEbRS+ILyt3L+nIkl02Km I+1XPszo8pO3+7XSuLQBcuYN4HaBEzCONhmkIm0gXzO57kSAtq1b9j1xtKLyEzbW 9K0EjVvSBGszosEEjcunzcXV1ZqGKPRveCPhRQF5zbWhnic/9c9tE76IvKNLWjlo S5CM5JW+/KEbB1VoLsEz2UX1IIiyIFETgF2wjPGs2iW+7TAXJX+o9+YGKOH1YV3W lJGndSWaCjaek1PERC56OOzh3lSdcLXflAUPMap5h/wAaQvNB0gYtKcpgyLB9kGf Neb0745hHBFdBnOssuyCTEqU4DC6DS3mbF83a3wJuXM5Y8Xh8nZe5c1G+BBneQYL yrxbaPfTvFHWbsmDi0MVQUAxeZOi/NEJalsSNfqU+BToOfne/NU= =9iba -----END PGP SIGNATURE----- --=-=-=--