From: ludo@gnu.org (Ludovic Courtès)
To: Andy Wingo <wingo@igalia.com>
Cc: guile-devel <guile-devel@gnu.org>
Subject: Re: Compiler memory consumption
Date: Mon, 30 Oct 2017 16:52:05 +0100 [thread overview]
Message-ID: <87376063bu.fsf@gnu.org> (raw)
In-Reply-To: <87a808mzsy.fsf@igalia.com> (Andy Wingo's message of "Mon, 30 Oct 2017 16:16:13 +0100")
Hello,
Andy Wingo <wingo@igalia.com> skribis:
> On Wed 25 Oct 2017 19:42, ludo@gnu.org (Ludovic Courtès) writes:
>
>> In short, compiling the top-level thunk is what’s killing us, because
>> the space and time complexity is proportional to the number of labels
>> therein.
>>
>> Also, our 16K line python.scm file translates into 428K labels, which
>> during slot allocation translates into a dozen of 428K-element intmaps
>> and intsets. So the compilation cost is space per source line of code
>> is high.
>>
>> Andy, what are your thoughts?
>
> I think that probably these 428K labels are partitioned into a number of
> functions. It is true that the biggest one takes the most time. Should
> we attempt to speed this up, or should we try harder to simplify this
> graph even at low optimization levels (e.g. "simplify" pass), or should
> we avoid CSE and periodically split liveness ranges, or should we use a
> different register allocation strategy on large functions?
The space taken by the data structures used to represent functions and
live variables is huge: we’re at 1+ GiB heap after the
‘compute-live-variables’ call for the big top-level function (compiling
the 4K subsequent functions, which are small, does not increase heap
size and is very fast). I think it’s primarily a space complexity
issue.
Ideally memory consumption would not be proportional to the number of
definitions in a file (top-level statements). Periodically splitting
liveness ranges could probably help avoid this pathological behavior: it
would place an upper bound on memory consumption.
Hopefully this wouldn’t have any noticeable impact on the generated
code, definitely not for top-level functions like that one.
WDYT?
I’m not sure what it takes to implement this splitting, though!
> Or would a transversal solution like a JIT actually be the solution?
> Or does the stack marking show the same pessimal behavior as the weak
> table, in that it's a large object behind a mark function?
My impression is that these are secondary issues.
Thanks for your feedback!
Ludo’.
prev parent reply other threads:[~2017-10-30 15:52 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-16 16:19 Compiler memory consumption Ludovic Courtès
2017-05-16 20:14 ` Andy Wingo
2017-05-16 20:45 ` Ludovic Courtès
2017-09-12 9:06 ` Ludovic Courtès
2017-09-17 13:56 ` Weak tables harmful to GC? Ludovic Courtès
2017-10-09 13:10 ` Christopher Allan Webber
2017-10-22 1:52 ` Ludovic Courtès
2017-10-22 2:20 ` Christopher Allan Webber
2017-10-23 0:16 ` Ludovic Courtès
2017-10-24 15:02 ` Ricardo Wurmus
2017-10-24 15:32 ` Ricardo Wurmus
2017-10-25 6:38 ` Ricardo Wurmus
2017-10-26 7:03 ` Ludovic Courtès
2017-10-26 8:35 ` Ricardo Wurmus
2017-10-26 16:52 ` Ricardo Wurmus
2017-10-27 5:28 ` Ludovic Courtès
2017-10-28 9:56 ` Ricardo Wurmus
2017-10-30 12:35 ` Ludovic Courtès
2017-10-30 14:48 ` Ricardo Wurmus
2017-10-30 17:20 ` Ricardo Wurmus
2017-10-30 22:18 ` Ludovic Courtès
2017-10-30 17:29 ` bug#19180: " Andy Wingo
2017-10-30 23:13 ` Ludovic Courtès
2017-10-31 8:25 ` Andy Wingo
2017-10-31 16:56 ` Ludovic Courtès
2017-10-26 17:17 ` Ludovic Courtès
2017-10-24 22:45 ` Christopher Allan Webber
2017-10-24 22:58 ` Ludovic Courtès
2017-10-25 0:50 ` Christopher Allan Webber
2017-10-25 17:11 ` Ludovic Courtès
2017-10-25 17:42 ` Compiler memory consumption Ludovic Courtès
2017-10-30 15:16 ` Andy Wingo
2017-10-30 15:52 ` Ludovic Courtès [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87376063bu.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=guile-devel@gnu.org \
--cc=wingo@igalia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).