Compilation speed

[Lars Ingebrigtsen <larsi@gnus.org>]

A nice thing about new, shiny machines with a bunch of cores is that it
makes saying "make bootstrap" faster when I want to check stuff.

However, there's a couple of sections that are single-threaded, which
makes the machine mostly idle while building, and I wondered whether
anybody has looked into these things...

I did a bootstrap (with make -j8) and timestamped the lines to see
whether anything in particular stands out.

[larsi@stories ~/src/emacs/trunk]$ emake bootstrap 2>&1 | ts -s
00:00:00 rm -f libXMenu11.a ./*.o deps/*
...

First deleting a lot of stuff...  that's fine...

00:00:03 running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion
00:00:03 checking for xcrun... no
00:00:03 checking for GNU Make... make

[...]

00:00:22 config.status: executing etc-refcards-emacsver.tex commands

And then ./configure takes 19 seconds, and that's fully single-threaded,
I believe?  And...  I'm guessing there's no way to get that to be
multi-threaded?

But I've also long wondered whether anybody has actually pruned the
tests lately?  Are all those tests (for things that seem really trivial,
mostly) really necessary these days?

And then compilation of Emacs and the manual starts, and that bit uses
all eight cored:

00:00:22   GEN      alloca.h
[...]
00:00:22   GEN      ../../info/eintr.info
00:00:22   GEN      sys/random.h
00:00:22   GEN      sys/select.h
00:00:22   GEN      ../../info/elisp.info
00:00:38   GEN      sys/stat.h
00:00:38   GEN      sys/time.h
00:00:38   GEN      sys/types.h
[...]
00:00:50   CC       minibuLoading loadup.el (source)...
00:00:50 dump mode: pbootstrap

So the build of the temacs binary just takes 28 seconds.

00:00:50 Loading emacs-lisp/byte-run (source)...
00:00:50 Loading emacs-lisp/backquote (source)...
[...]
00:01:02 Dump complete

And dumping takes 12 seconds.  Then some other bits are compiled and
then we start the pbootstrap:

00:01:07 ./temacs --batch  -l loadup --temacs=pbootstrap

This seems to proceed in parallell until we reach:

00:01:17   INFO     Scraping files for loaddefs.el... 
[...]
00:01:54   INFO     Scraping files for loaddefs.el...done

Which is single-threaded and takes a whopping 37 seconds!  Could that
possibly be parallelised in some way?  Probably be tricky...

00:01:54 Loading loadup.el (source)...
[...]
00:01:58 Dump complete

Another dump; takes just four seconds.

00:02:00   ELC      ../lisp/language/khmer.elc
00:02:00   ELC      ../lisp/language/korean.elc
[...]
00:02:46   ELC      progmodes/cc-styles.elc
00:02:46   ELC      progmodes/cc-mode.elc

And finally, all the remaining (undumped) .el files are compiled, and
this uses all the cores, so it takes just 46 seconds.

Hm...  well, I guess if somebody wanted to speed up compilation, then
the loaddefs scraping is the biggest single item here to look at.

But to sum up, (+ 22 12 37 4) => 75 seconds of the Emacs compilation is
single-threaded, while 90 seconds are multi-threaded (and as machines
get more and more cores, we'll probably see the single-threaded parts
take more than 50% of the time spent).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no