Emacs build times on Windows

Emacs build times on Windows

[Christoph Scholtes]

Hi,

My personal Jenkins build server tells me that building Emacs on Windows 
from a clean trunk via `make bootstrap', `make info', `make install-bin' 
takes around 50 minutes on a fairly recent Centrino Dual Core Laptop.

Granted that we are doing (unnecessary) rebuilds of code with PURESIZE 
(which I am trying to eliminate) it is still ridiculously slow.

Especially the byte compilation is crawling compared to a GNU/Linux 
machine. At work I have a decent quad core desktop with GNU/Linux and I 
can kick off a bootstrap with -j4 and it is done before I come back from 
getting my coffee. Especially byte compilation stage seems to be a lot 
faster on GNU/Linux than it is on Windows.

Any ideas why?

Christoph

Re: Emacs build times on Windows

[Eli Zaretskii]

> Date: Fri, 20 Apr 2012 21:50:42 -0600
> From: Christoph Scholtes <cschol2112@googlemail.com>
> 
> My personal Jenkins build server tells me that building Emacs on Windows 
> from a clean trunk via `make bootstrap', `make info', `make install-bin' 
> takes around 50 minutes on a fairly recent Centrino Dual Core Laptop.

On my i7 desktop (4 hyper-threaded cores), it takes 9 minutes, if I
use the -j10 switch to Make.  I suggest that you try -j6 in your
builds (if you use anything older than the current CVS of GNU Make,
you will also have to use the XMFLAGS="-j6" kludge as well, see
nt/INSTALL).  A parallel build should be faster.

> Granted that we are doing (unnecessary) rebuilds of code with PURESIZE 
> (which I am trying to eliminate) it is still ridiculously slow.

The extra build with a large PURESIZE is not the problem.  Each one of
the 2 compilations of the Emacs sources takes 20 seconds on my machine
(using "make -j10"), which is negligibly short compared to the
compilation of Lisp files.  ("make -j10 info" takes another 5 seconds,
but only on the trunk, where I recently made the 'info' target of
nt/makefile.w32-in more parallel-friendly.)

> Especially the byte compilation is crawling compared to a GNU/Linux 
> machine. At work I have a decent quad core desktop with GNU/Linux and I 
> can kick off a bootstrap with -j4 and it is done before I come back from 
> getting my coffee. Especially byte compilation stage seems to be a lot 
> faster on GNU/Linux than it is on Windows.
> 
> Any ideas why?

Very simple: the byte compilation on Posix platforms can run in
parallel, because it uses a generic .el.elc rule.  By contrast, on
Windows we compile them sequentially, one after the other, because the
recipe says:

  compile-CMD:
  #	-for %%f in ($(lisp) $(WINS)) do for %%g in (%%f\*.elc) do @attrib -r %%g
	  for %%f in ($(COMPILE_FIRST)) do \
	    $(emacs) -l loaddefs $(BYTE_COMPILE_FLAGS) -f batch-byte-compile-if-not-done %%f
	  for %%f in (. $(WINS)) do for %%g in (%%f/*.el) do \
	    $(emacs) -l loaddefs $(BYTE_COMPILE_FLAGS) -f batch-byte-compile-if-not-done %%f/%%g

IOW, we use a shell-level loop to compile the files one by one.

So if you are looking for the place to significantly speed up the
bootstrap, the first place to look is to make the byte compilation use
a similar setup to what you see in lisp/Makefile.in.

Re: Emacs build times on Windows

[Christoph Scholtes]

Hi Eli,

On 4/21/2012 1:29 AM, Eli Zaretskii wrote:

> On my i7 desktop (4 hyper-threaded cores), it takes 9 minutes, if I
> use the -j10 switch to Make.  I suggest that you try -j6 in your
> builds (if you use anything older than the current CVS of GNU Make,
> you will also have to use the XMFLAGS="-j6" kludge as well, see
> nt/INSTALL).  A parallel build should be faster.

Thanks. Does this work with mingw32-make also or do I have to use Cygwin 
(or MSYS?) to obtain gmake?

I tried building with mingw32-make -j2 but it did not result in any 
significant speedup.

> The extra build with a large PURESIZE is not the problem.  Each one of
> the 2 compilations of the Emacs sources takes 20 seconds on my machine
> (using "make -j10"), which is negligibly short compared to the
> compilation of Lisp files.  ("make -j10 info" takes another 5 seconds,
> but only on the trunk, where I recently made the 'info' target of
> nt/makefile.w32-in more parallel-friendly.)

On my system the extra PURESIZE build takes a lot longer than 20 
seconds, but hopefully I can reduce that by using parallel compilation.

> Very simple: the byte compilation on Posix platforms can run in
> parallel, because it uses a generic .el.elc rule.  By contrast, on
> Windows we compile them sequentially, one after the other, because the
> recipe says:
>
>    compile-CMD:
>    #	-for %%f in ($(lisp) $(WINS)) do for %%g in (%%f\*.elc) do @attrib -r %%g
> 	  for %%f in ($(COMPILE_FIRST)) do \
> 	    $(emacs) -l loaddefs $(BYTE_COMPILE_FLAGS) -f batch-byte-compile-if-not-done %%f
> 	  for %%f in (. $(WINS)) do for %%g in (%%f/*.el) do \
> 	    $(emacs) -l loaddefs $(BYTE_COMPILE_FLAGS) -f batch-byte-compile-if-not-done %%f/%%g
>
> IOW, we use a shell-level loop to compile the files one by one.
>
> So if you are looking for the place to significantly speed up the
> bootstrap, the first place to look is to make the byte compilation use
> a similar setup to what you see in lisp/Makefile.in.

I see. I will look into that. Thanks.

Christoph

Re: Emacs build times on Windows

[Eli Zaretskii]

> Date: Sat, 21 Apr 2012 06:05:42 -0600
> From: Christoph Scholtes <cschol2112@googlemail.com>
> CC: emacs-devel@gnu.org
> 
> > On my i7 desktop (4 hyper-threaded cores), it takes 9 minutes, if I
> > use the -j10 switch to Make.  I suggest that you try -j6 in your
> > builds (if you use anything older than the current CVS of GNU Make,
> > you will also have to use the XMFLAGS="-j6" kludge as well, see
> > nt/INSTALL).  A parallel build should be faster.
> 
> Thanks. Does this work with mingw32-make also or do I have to use Cygwin 
> (or MSYS?) to obtain gmake?

Parallel builds are supported by the native Windows build of GNU Make
since v3.81.  So if you have mingw32-make v3.81 or 3.82, you should be
all set for parallelism.

> I tried building with mingw32-make -j2 but it did not result in any 
> significant speedup.

Like I said: you need to say "mingw32-make -j2 XMFLAGS=-j2" for this
to work.

The issue here is that the native Windows build of Make does not yet
support the "job server" method of parallel execution, which causes
the number of jobs in each sub-Make be reset to 1.  The XMFLAGS=-j2
kludge works around that by forcing the sub-Make's to use an explicit
"-jN" switch.

(The current CVS code of Make includes a job server implementation for
Windows, so the next release of Make will free us from this kludge.)

> > IOW, we use a shell-level loop to compile the files one by one.
> >
> > So if you are looking for the place to significantly speed up the
> > bootstrap, the first place to look is to make the byte compilation use
> > a similar setup to what you see in lisp/Makefile.in.
> 
> I see. I will look into that. Thanks.

Thanks in advance.