From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Newsgroups: gmane.lisp.guile.bugs Subject: bug#19180: Weak tables harmful to GC? Date: Tue, 31 Oct 2017 17:56:52 +0100 Message-ID: <87zi87w90r.fsf__9495.81585960022$1509469045$gmane$org@gnu.org> References: <87wp9gwz8m.fsf@gnu.org> <87fug4o8z2.fsf@pobox.com> <87wp54z3p5.fsf@gnu.org> <87zi9th1i6.fsf_-_@gnu.org> <87y3o454pr.fsf@gnu.org> <87r2tvncs5.fsf@dustycloud.org> <87wp3mwwef.fsf@gnu.org> <87mv4gd0ik.fsf@elephly.net> <87a80eie63.fsf@gnu.org> <87k1zimhmt.fsf@elephly.net> <87bmktn96e.fsf@elephly.net> <87tvyl9n22.fsf@gnu.org> <87r2tnlhno.fsf@elephly.net> <87a8087qz7.fsf@gnu.org> <87she0lf1y.fsf@igalia.com> <87zi88ut3k.fsf@gnu.org> <877evbiuzy.fsf@igalia.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: blaine.gmane.org 1509469045 17443 195.159.176.226 (31 Oct 2017 16:57:25 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 31 Oct 2017 16:57:25 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) Cc: 19180@debbugs.gnu.org, Ricardo Wurmus , guile-devel@gnu.org To: Andy Wingo Original-X-From: bug-guile-bounces+guile-bugs=m.gmane.org@gnu.org Tue Oct 31 17:57:19 2017 Return-path: Envelope-to: guile-bugs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1e9Zqj-0003Xg-9A for guile-bugs@m.gmane.org; Tue, 31 Oct 2017 17:57:13 +0100 Original-Received: from localhost ([::1]:46610 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e9Zqq-0002eT-OT for guile-bugs@m.gmane.org; Tue, 31 Oct 2017 12:57:20 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:36799) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e9Zqc-0002ap-Lh for bug-guile@gnu.org; Tue, 31 Oct 2017 12:57:12 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e9ZqY-0001EZ-CQ for bug-guile@gnu.org; Tue, 31 Oct 2017 12:57:06 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:35768) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1e9ZqY-0001ED-8M for bug-guile@gnu.org; Tue, 31 Oct 2017 12:57:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1e9ZqX-0007JV-TY for bug-guile@gnu.org; Tue, 31 Oct 2017 12:57:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Tue, 31 Oct 2017 16:57:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 19180 X-GNU-PR-Package: guile X-GNU-PR-Keywords: Original-Received: via spool by 19180-submit@debbugs.gnu.org id=B19180.150946901828104 (code B ref 19180); Tue, 31 Oct 2017 16:57:01 +0000 Original-Received: (at 19180) by debbugs.gnu.org; 31 Oct 2017 16:56:58 +0000 Original-Received: from localhost ([127.0.0.1]:44449 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1e9ZqU-0007JE-1u for submit@debbugs.gnu.org; Tue, 31 Oct 2017 12:56:58 -0400 Original-Received: from hera.aquilenet.fr ([141.255.128.1]:42031) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1e9ZqS-0007J5-Ar for 19180@debbugs.gnu.org; Tue, 31 Oct 2017 12:56:56 -0400 Original-Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id CA31AFAA0; Tue, 31 Oct 2017 17:56:55 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Original-Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3wXrxNkDQLCe; Tue, 31 Oct 2017 17:56:54 +0100 (CET) Original-Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 8F424F36D; Tue, 31 Oct 2017 17:56:54 +0100 (CET) X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 10 Brumaire an 226 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu In-Reply-To: <877evbiuzy.fsf@igalia.com> (Andy Wingo's message of "Tue, 31 Oct 2017 09:25:53 +0100") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-guile@gnu.org List-Id: "Bug reports for GUILE, GNU's Ubiquitous Extension Language" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guile-bounces+guile-bugs=m.gmane.org@gnu.org Original-Sender: "bug-guile" Xref: news.gmane.org gmane.lisp.guile.bugs:8878 Archived-At: Heya! Andy Wingo skribis: > On Tue 31 Oct 2017 00:13, ludo@gnu.org (Ludovic Court=C3=A8s) writes: > >> I built libguile with the patch (I haven=E2=80=99t yet taken the time to= rebuild >> all of Guile). It works, but unfortunately it still shows quick growth >> of the heap on this example (): > > Fixed in attached patches (on top of the other one). This was a race > between the periodic vacuum process, which should run after GC via a > finalizer, and the mutator. If the mutator had the weak table lock, the > vacuum would never be run. Of course in the test below, the mutator > usually has the table lock, so we ended up often skipping the vacuum, > which causes the table size to grow, which causes the active heap size > to grow, which causes the bytes-allocated-before-GC to increase, and > which ultimately is a vicious circle. > > In my tests this patch fixes the issue, though the level at which the > heap stabilizes can vary slightly as there's a bit of nondeterministic > concurrency as the mutator and the vacuum process still race a bit. > Right now for me it stabilizes at about 6.2M of heap. Yes. I can confirm that it fixes this issue. One of my benchmarks is to =E2=80=98read=E2=80=99 the 26M file that contain= s the CPS-as-sexps representation of gnu/packages/python.scm, with source-location recording on (thus with a big source property table). Compared to 2.2.2, execution time is almost divided by two; heap size is usually 540M and 620M, whereas it varies between 560M and 920M with 2.2.2: --8<---------------cut here---------------start------------->8--- ludo@ribbon ~/src/guix$ guile profile-cps-read.scm clock utime stime cutime cstime gctime 8.23 10.00 0.06 0.00 0.00 5.17 ;;; (#) with Guile 2.2.2: heap-size from 2.53 MiB to 780.03 MiB ludo@ribbon ~/src/guix$ guile profile-cps-read.scm clock utime stime cutime cstime gctime 8.29 9.94 0.04 0.00 0.00 4.97 ;;; (#) with Guile 2.2.2: heap-size from 2.53 MiB to 660.36 MiB ludo@ribbon ~/src/guix$ guile profile-cps-read.scm clock utime stime cutime cstime gctime 8.11 9.33 0.05 0.00 0.00 3.90 ;;; (#) with Guile 2.2.2: heap-size from 2.53 MiB to 562.21 MiB ludo@ribbon ~/src/guix$ guile profile-cps-read.scm clock utime stime cutime cstime gctime 8.23 9.90 0.08 0.00 0.00 5.22 ;;; (#) with Guile 2.2.2: heap-size from 2.53 MiB to 918.09 MiB ludo@ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm clock utime stime cutime cstime gctime 4.68 6.71 0.02 0.00 0.00 3.88 ;;; (#) with Guile 2.2.2.17-8069-dirty: heap-size from 1.89 MiB to 540.54 MiB ludo@ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm clock utime stime cutime cstime gctime 4.73 6.77 0.05 0.00 0.00 4.03 ;;; (#) with Guile 2.2.2.17-8069-dirty: heap-size from 1.89 MiB to 620.22 MiB ludo@ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm clock utime stime cutime cstime gctime 4.69 6.80 0.06 0.00 0.00 3.94 ;;; (#) with Guile 2.2.2.17-8069-dirty: heap-size from 1.89 MiB to 573.36 MiB ludo@ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm clock utime stime cutime cstime gctime 4.67 6.81 0.06 0.00 0.00 4.05 ;;; (#) with Guile 2.2.2.17-8069-dirty: heap-size from 1.89 MiB to 547.39 MiB --8<---------------cut here---------------end--------------->8--- Another benchmark is the =E2=80=98emit-bytecode=E2=80=99 procedure for this= CPS, without optimizations turned off (like =E2=80=98%lightweight-optimizations=E2=80=99= in Guix.) The heap size after the =E2=80=98emit-bytecode=E2=80=99 call is still at 1.= 3G (338M before the call), about the same as with 2.2.2. That=E2=80=99s not surpris= ing because I think memory consumption comes from the data structures used at that compilation stage, as discussed before. The wall-clock time is ~45s, whereas it=E2=80=99s ~54s with 2.2.2. In other news, I=E2=80=99ve rebuilt 2.2.2 + these patches with Guix, and everything went fine (Guile processes seem to peak at ~150M resident when compiling). So, all the lights are green on my side! Thanks a whole lot for coming up with this solution! Ludo=E2=80=99.