* Speeding up parallel compilation
@ 2012-09-06 3:13 Stefan Monnier
2012-09-06 12:32 ` joakim
2012-09-06 16:19 ` parallel compilation --with-ns (was: re: Speeding up parallel compilation) chad
0 siblings, 2 replies; 7+ messages in thread
From: Stefan Monnier @ 2012-09-06 3:13 UTC (permalink / raw)
To: emacs-devel
The recent change I installed which macro-expands Elisp code eagerly
upon load does not speed up sequential compilation very much, but it
seems to help parallel compilation significantly: on a 12-core
24-threads machine, "make -j24 bootstrap" went down from 2:20 to 1:50,
and the CPU utilization went up (from 800% to 1000%).
In other words, this mostly sped up a part of the compilation which does
not parallelize well. I guess that would be either the construction of
loaddefs, or the compilation of the compiler.
So, lucky those with manycore systems.
Stefan
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Speeding up parallel compilation
2012-09-06 3:13 Speeding up parallel compilation Stefan Monnier
@ 2012-09-06 12:32 ` joakim
2012-09-06 14:30 ` Stefan Monnier
2012-09-06 16:19 ` parallel compilation --with-ns (was: re: Speeding up parallel compilation) chad
1 sibling, 1 reply; 7+ messages in thread
From: joakim @ 2012-09-06 12:32 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
Stefan Monnier <monnier@iro.umontreal.ca> writes:
> The recent change I installed which macro-expands Elisp code eagerly
> upon load does not speed up sequential compilation very much, but it
> seems to help parallel compilation significantly: on a 12-core
> 24-threads machine, "make -j24 bootstrap" went down from 2:20 to 1:50,
> and the CPU utilization went up (from 800% to 1000%).
>
> In other words, this mostly sped up a part of the compilation which does
> not parallelize well. I guess that would be either the construction of
> loaddefs, or the compilation of the compiler.
>
> So, lucky those with manycore systems.
Yay, thus I must get one! BTW was that a dual AMD or Intel btw?
>
>
> Stefan
--
Joakim Verona
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Speeding up parallel compilation
2012-09-06 12:32 ` joakim
@ 2012-09-06 14:30 ` Stefan Monnier
0 siblings, 0 replies; 7+ messages in thread
From: Stefan Monnier @ 2012-09-06 14:30 UTC (permalink / raw)
To: joakim; +Cc: emacs-devel
> Yay, thus I must get one! BTW was that a dual AMD or Intel btw?
I think it's an Intel rig (based on the "12-cores vs 24-threads" and the
fact that I don't think AMD has multithreaded CPUs (they seem to prefer
sharing the FPU part (and sometimes the decoder) between 2 cores
instead)).
[ And the times are with
"--enable-checking --enable-check-lisp-object-type". ]
Sadly, the machines I use most of the time are pretty far from that,
using low power net(book|top) CPUs, so bootstrapping takes more like
half an hour (so I work hard to avoid bootstrapping).
Stefan
^ permalink raw reply [flat|nested] 7+ messages in thread
* parallel compilation --with-ns (was: re: Speeding up parallel compilation)
2012-09-06 3:13 Speeding up parallel compilation Stefan Monnier
2012-09-06 12:32 ` joakim
@ 2012-09-06 16:19 ` chad
2012-09-06 16:21 ` parallel compilation --with-ns Paul Eggert
2012-09-10 18:25 ` Sudish Joseph
1 sibling, 2 replies; 7+ messages in thread
From: chad @ 2012-09-06 16:19 UTC (permalink / raw)
To: emacs-devel@gnu.org discussions
On 05 Sep 2012, at 20:13, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> The recent change [...] seems to help parallel compilation significantly:
Under macosx (i.e. --with-ns), I've tried using 'make -j4' off
and on over the years, and frequently run into problems during
the build. I end up without a complete nextstep/Emacs.app most of
the time, and it seems to be a problem with directories not being
created before one of the workers wants to use it. Three
questions:
1.) Is the ns-app code known to be unable to use parallel
compilation?
2.) Does it matter that I do full rebuilds each time, starting
with autogen.sh and ending with make bootstrap install)?
3.) Can anyone suggest a good way to start debugging this?
Thanks,
*Chad
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: parallel compilation --with-ns
2012-09-06 16:19 ` parallel compilation --with-ns (was: re: Speeding up parallel compilation) chad
@ 2012-09-06 16:21 ` Paul Eggert
2012-09-10 18:25 ` Sudish Joseph
1 sibling, 0 replies; 7+ messages in thread
From: Paul Eggert @ 2012-09-06 16:21 UTC (permalink / raw)
To: chad; +Cc: emacs-devel@gnu.org discussions
On 09/06/2012 09:19 AM, chad wrote:
> 3.) Can anyone suggest a good way to start debugging this?
For stuff like this I use "truss" (or its equivalent)
and "make -d". It's not fun, but sometimes it works.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: parallel compilation --with-ns
2012-09-06 16:19 ` parallel compilation --with-ns (was: re: Speeding up parallel compilation) chad
2012-09-06 16:21 ` parallel compilation --with-ns Paul Eggert
@ 2012-09-10 18:25 ` Sudish Joseph
2012-09-10 20:41 ` chad
1 sibling, 1 reply; 7+ messages in thread
From: Sudish Joseph @ 2012-09-10 18:25 UTC (permalink / raw)
To: chad; +Cc: emacs-devel@gnu.org discussions
chad <yandros@MIT.EDU> writes:
> Under macosx (i.e. --with-ns), I've tried using 'make -j4' off
> and on over the years, and frequently run into problems during
> the build. I end up without a complete nextstep/Emacs.app most of
> the time, and it seems to be a problem with directories not being
> created before one of the workers wants to use it. Three
> questions:
>
> 1.) Is the ns-app code known to be unable to use parallel
> compilation?
I was doing regular make -j8 builds of the ns port up until a few months
ago and had no problems with it.
> 2.) Does it matter that I do full rebuilds each time, starting
> with autogen.sh and ending with make bootstrap install)?
Here's the essence of the function I use to build the ns port.
sj_emacsbuild () {
local build_dir
rm -rf sj-build.*
build_dir=sj-build.$$
sh ./autogen.sh
mkdir $build_dir || {
echo "couldn't mkdir $build_dir: $!"
return 1
}
cd $build_dir
env "$EMACS_CC_FLAGS[@]" ../configure "$EMACS_CONF_FLAGS[@]" && nice -n 20 make -j8 && nice -n 20 make install
}
I then copy over Emacs.app to ~/Applications using a separate func.
You don't need to do a explicit make bootstrap, iirc.
"make maintainer-clean && make -j8" does a full build.
I used to have "git clean -xfd" in the build func above but didn't
adjust for that when switching to the bzr tree. I.e., full builds
should work in parallel with the ns port.
-Sudish
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: parallel compilation --with-ns
2012-09-10 18:25 ` Sudish Joseph
@ 2012-09-10 20:41 ` chad
0 siblings, 0 replies; 7+ messages in thread
From: chad @ 2012-09-10 20:41 UTC (permalink / raw)
To: emacs-devel@gnu.org discussions; +Cc: Sudish Joseph
On 10 Sep 2012, at 11:25, Sudish Joseph <sudish@gmail.com> wrote:
>> 1.) Is the ns-app code known to be unable to use parallel
>> compilation?
>
> I was doing regular make -j8 builds of the ns port up until a few months
> ago and had no problems with it.
FWIW, the key seems to be that the install target needs to wait for the
bootstrap or all target to complete (i.e. `make install' in a separate
process after `make -j8 all/bootstrap').
Thanks for the help, everyone.
*Chad
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2012-09-10 20:41 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-09-06 3:13 Speeding up parallel compilation Stefan Monnier
2012-09-06 12:32 ` joakim
2012-09-06 14:30 ` Stefan Monnier
2012-09-06 16:19 ` parallel compilation --with-ns (was: re: Speeding up parallel compilation) chad
2012-09-06 16:21 ` parallel compilation --with-ns Paul Eggert
2012-09-10 18:25 ` Sudish Joseph
2012-09-10 20:41 ` chad
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).