From: Andy Wingo <wingo@pobox.com>
To: ludo@gnu.org (Ludovic Courtès)
Cc: guile-devel <guile-devel@gnu.org>
Subject: Re: Compiler memory consumption
Date: Tue, 16 May 2017 22:14:09 +0200 [thread overview]
Message-ID: <87fug4o8z2.fsf@pobox.com> (raw)
In-Reply-To: <87wp9gwz8m.fsf@gnu.org> ("Ludovic Courtès"'s message of "Tue, 16 May 2017 18:19:37 +0200")
Hi!
Definitely sounds like we have an issue we should deal with.
On Tue 16 May 2017 18:19, ludo@gnu.org (Ludovic Courtès) writes:
> here is the gcprof output
I find gcprof useful when I want to improve runtime by removing
allocation. However I don't find it useful when dealing with memory use
issues; ymmv of course.
> while compiling gnu/packages/python.scm, which defines 841 package
> objects (structs) with 5 times more thunks of the form (lambda ()
> value):
You mention later:
> Also, for reference, loading python.go peaks at 315M RSS:
>
> $ \time ./pre-inst-env guile -c '(use-modules (gnu packages python))'
> 0.18user 0.02system 0:00.18elapsed 112%CPU (0avgtext+0avgdata 315648maxresident)k
> 0inputs+0outputs (0major+7784minor)pagefaults 0swaps
But this I don't understand. If I do a ./pre-inst-env guile and then
load (gnu packages python), I get a 20MB heap size, a 35MB total private
dirty memory and 52MB clean shared memory. (Measured using smaps via
https://wingolog.org/pub/mem_usage.py). This is at commit
60c9e80444421c412ae3d0e7b4b224ef0e32947f.
I just built the "time" package and I see similar numbers here. I can
only think that the "time" package's numbers are bogus.
> time(1) reports a maximum resident set size of 3.8G (though I see
> something around 900MiB in ‘top’.)
The only way Guile's memory usage can shrink in practice is if it
recurses a lot on the stack and then returns those pages to the OS. I
don't think libgc will return pages to the OS (though I could be
wrong). So that would be a possibility to look into, if time can be
trusted.
> When compiling python.scm #:to 'cps, we end up with 1G max RSS in 6s.
Measured with time? If this is the case it could be that python.scm is
just a lot of code. Any compiler would take a lot if the IR size is 1
GB.
> The only conclusion I can draw is that cps-to-bytecode compilation seems
> to be responsible for most of the memory consumption.
This is possible but I am not there yet. I don't see why compiling this
file to CPS should cause memory usage of 1GB. That is 9000 memory words
per textual line -- simply too much.
Many unknowns here!
Andy
next prev parent reply other threads:[~2017-05-16 20:14 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-16 16:19 Compiler memory consumption Ludovic Courtès
2017-05-16 20:14 ` Andy Wingo [this message]
2017-05-16 20:45 ` Ludovic Courtès
2017-09-12 9:06 ` Ludovic Courtès
2017-09-17 13:56 ` Weak tables harmful to GC? Ludovic Courtès
2017-10-09 13:10 ` Christopher Allan Webber
2017-10-22 1:52 ` Ludovic Courtès
2017-10-22 2:20 ` Christopher Allan Webber
2017-10-23 0:16 ` Ludovic Courtès
2017-10-24 15:02 ` Ricardo Wurmus
2017-10-24 15:32 ` Ricardo Wurmus
2017-10-25 6:38 ` Ricardo Wurmus
2017-10-26 7:03 ` Ludovic Courtès
2017-10-26 8:35 ` Ricardo Wurmus
2017-10-26 16:52 ` Ricardo Wurmus
2017-10-27 5:28 ` Ludovic Courtès
2017-10-28 9:56 ` Ricardo Wurmus
2017-10-30 12:35 ` Ludovic Courtès
2017-10-30 14:48 ` Ricardo Wurmus
2017-10-30 17:20 ` Ricardo Wurmus
2017-10-30 22:18 ` Ludovic Courtès
2017-10-30 17:29 ` bug#19180: " Andy Wingo
2017-10-30 23:13 ` Ludovic Courtès
2017-10-31 8:25 ` Andy Wingo
2017-10-31 16:56 ` Ludovic Courtès
2017-10-26 17:17 ` Ludovic Courtès
2017-10-24 22:45 ` Christopher Allan Webber
2017-10-24 22:58 ` Ludovic Courtès
2017-10-25 0:50 ` Christopher Allan Webber
2017-10-25 17:11 ` Ludovic Courtès
2017-10-25 17:42 ` Compiler memory consumption Ludovic Courtès
2017-10-30 15:16 ` Andy Wingo
2017-10-30 15:52 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87fug4o8z2.fsf@pobox.com \
--to=wingo@pobox.com \
--cc=guile-devel@gnu.org \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).