From: ludo@gnu.org (Ludovic Courtès)
To: Andy Wingo <wingo@igalia.com>
Cc: 19180@debbugs.gnu.org,
Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>,
guile-devel@gnu.org
Subject: Re: Weak tables harmful to GC?
Date: Tue, 31 Oct 2017 17:56:52 +0100 [thread overview]
Message-ID: <87zi87w90r.fsf@gnu.org> (raw)
In-Reply-To: <877evbiuzy.fsf@igalia.com> (Andy Wingo's message of "Tue, 31 Oct 2017 09:25:53 +0100")
Heya!
Andy Wingo <wingo@igalia.com> skribis:
> On Tue 31 Oct 2017 00:13, ludo@gnu.org (Ludovic Courtès) writes:
>
>> I built libguile with the patch (I haven’t yet taken the time to rebuild
>> all of Guile). It works, but unfortunately it still shows quick growth
>> of the heap on this example (<https://bugs.gnu.org/28590>):
>
> Fixed in attached patches (on top of the other one). This was a race
> between the periodic vacuum process, which should run after GC via a
> finalizer, and the mutator. If the mutator had the weak table lock, the
> vacuum would never be run. Of course in the test below, the mutator
> usually has the table lock, so we ended up often skipping the vacuum,
> which causes the table size to grow, which causes the active heap size
> to grow, which causes the bytes-allocated-before-GC to increase, and
> which ultimately is a vicious circle.
>
> In my tests this patch fixes the issue, though the level at which the
> heap stabilizes can vary slightly as there's a bit of nondeterministic
> concurrency as the mutator and the vacuum process still race a bit.
> Right now for me it stabilizes at about 6.2M of heap.
Yes. I can confirm that it fixes this issue.
One of my benchmarks is to ‘read’ the 26M file that contains the
CPS-as-sexps representation of gnu/packages/python.scm, with
source-location recording on (thus with a big source property table).
Compared to 2.2.2, execution time is almost divided by two; heap size is
usually 540M and 620M, whereas it varies between 560M and 920M with
2.2.2:
--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ guile profile-cps-read.scm
clock utime stime cutime cstime gctime
8.23 10.00 0.06 0.00 0.00 5.17
;;; (#<weak-table 1863537/3595271>)
with Guile 2.2.2:
heap-size from 2.53 MiB to 780.03 MiB
ludo@ribbon ~/src/guix$ guile profile-cps-read.scm
clock utime stime cutime cstime gctime
8.29 9.94 0.04 0.00 0.00 4.97
;;; (#<weak-table 1820831/3595271>)
with Guile 2.2.2:
heap-size from 2.53 MiB to 660.36 MiB
ludo@ribbon ~/src/guix$ guile profile-cps-read.scm
clock utime stime cutime cstime gctime
8.11 9.33 0.05 0.00 0.00 3.90
;;; (#<weak-table 1829710/3595271>)
with Guile 2.2.2:
heap-size from 2.53 MiB to 562.21 MiB
ludo@ribbon ~/src/guix$ guile profile-cps-read.scm
clock utime stime cutime cstime gctime
8.23 9.90 0.08 0.00 0.00 5.22
;;; (#<weak-table 1828628/3595271>)
with Guile 2.2.2:
heap-size from 2.53 MiB to 918.09 MiB
ludo@ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm
clock utime stime cutime cstime gctime
4.68 6.71 0.02 0.00 0.00 3.88
;;; (#<weak-table 1799062/3595271>)
with Guile 2.2.2.17-8069-dirty:
heap-size from 1.89 MiB to 540.54 MiB
ludo@ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm
clock utime stime cutime cstime gctime
4.73 6.77 0.05 0.00 0.00 4.03
;;; (#<weak-table 1801035/3595271>)
with Guile 2.2.2.17-8069-dirty:
heap-size from 1.89 MiB to 620.22 MiB
ludo@ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm
clock utime stime cutime cstime gctime
4.69 6.80 0.06 0.00 0.00 3.94
;;; (#<weak-table 1796895/3595271>)
with Guile 2.2.2.17-8069-dirty:
heap-size from 1.89 MiB to 573.36 MiB
ludo@ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm
clock utime stime cutime cstime gctime
4.67 6.81 0.06 0.00 0.00 4.05
;;; (#<weak-table 1797102/3595271>)
with Guile 2.2.2.17-8069-dirty:
heap-size from 1.89 MiB to 547.39 MiB
--8<---------------cut here---------------end--------------->8---
Another benchmark is the ‘emit-bytecode’ procedure for this CPS, without
optimizations turned off (like ‘%lightweight-optimizations’ in Guix.)
The heap size after the ‘emit-bytecode’ call is still at 1.3G (338M
before the call), about the same as with 2.2.2. That’s not surprising
because I think memory consumption comes from the data structures used
at that compilation stage, as discussed before.
The wall-clock time is ~45s, whereas it’s ~54s with 2.2.2.
In other news, I’ve rebuilt 2.2.2 + these patches with Guix, and
everything went fine (Guile processes seem to peak at ~150M resident
when compiling).
So, all the lights are green on my side!
Thanks a whole lot for coming up with this solution!
Ludo’.
next prev parent reply other threads:[~2017-10-31 16:56 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-16 16:19 Compiler memory consumption Ludovic Courtès
2017-05-16 20:14 ` Andy Wingo
2017-05-16 20:45 ` Ludovic Courtès
2017-09-12 9:06 ` Ludovic Courtès
2017-09-17 13:56 ` Weak tables harmful to GC? Ludovic Courtès
2017-10-09 13:10 ` Christopher Allan Webber
2017-10-22 1:52 ` Ludovic Courtès
2017-10-22 2:20 ` Christopher Allan Webber
2017-10-23 0:16 ` Ludovic Courtès
2017-10-24 15:02 ` Ricardo Wurmus
2017-10-24 15:32 ` Ricardo Wurmus
2017-10-25 6:38 ` Ricardo Wurmus
2017-10-26 7:03 ` Ludovic Courtès
2017-10-26 8:35 ` Ricardo Wurmus
2017-10-26 16:52 ` Ricardo Wurmus
2017-10-27 5:28 ` Ludovic Courtès
2017-10-28 9:56 ` Ricardo Wurmus
2017-10-30 12:35 ` Ludovic Courtès
2017-10-30 14:48 ` Ricardo Wurmus
2017-10-30 17:20 ` Ricardo Wurmus
2017-10-30 22:18 ` Ludovic Courtès
2017-10-30 17:29 ` bug#19180: " Andy Wingo
2017-10-30 23:13 ` Ludovic Courtès
2017-10-31 8:25 ` Andy Wingo
2017-10-31 16:56 ` Ludovic Courtès [this message]
2017-10-26 17:17 ` Ludovic Courtès
2017-10-24 22:45 ` Christopher Allan Webber
2017-10-24 22:58 ` Ludovic Courtès
2017-10-25 0:50 ` Christopher Allan Webber
2017-10-25 17:11 ` Ludovic Courtès
2017-10-25 17:42 ` Compiler memory consumption Ludovic Courtès
2017-10-30 15:16 ` Andy Wingo
2017-10-30 15:52 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87zi87w90r.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=19180@debbugs.gnu.org \
--cc=guile-devel@gnu.org \
--cc=ricardo.wurmus@mdc-berlin.de \
--cc=wingo@igalia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).