Compilation speed

Compilation speed

[Lars Ingebrigtsen]

A nice thing about new, shiny machines with a bunch of cores is that it
makes saying "make bootstrap" faster when I want to check stuff.

However, there's a couple of sections that are single-threaded, which
makes the machine mostly idle while building, and I wondered whether
anybody has looked into these things...

I did a bootstrap (with make -j8) and timestamped the lines to see
whether anything in particular stands out.

[larsi@stories ~/src/emacs/trunk]$ emake bootstrap 2>&1 | ts -s
00:00:00 rm -f libXMenu11.a ./*.o deps/*
...

First deleting a lot of stuff...  that's fine...

00:00:03 running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion
00:00:03 checking for xcrun... no
00:00:03 checking for GNU Make... make

[...]

00:00:22 config.status: executing etc-refcards-emacsver.tex commands

And then ./configure takes 19 seconds, and that's fully single-threaded,
I believe?  And...  I'm guessing there's no way to get that to be
multi-threaded?

But I've also long wondered whether anybody has actually pruned the
tests lately?  Are all those tests (for things that seem really trivial,
mostly) really necessary these days?

And then compilation of Emacs and the manual starts, and that bit uses
all eight cored:

00:00:22   GEN      alloca.h
[...]
00:00:22   GEN      ../../info/eintr.info
00:00:22   GEN      sys/random.h
00:00:22   GEN      sys/select.h
00:00:22   GEN      ../../info/elisp.info
00:00:38   GEN      sys/stat.h
00:00:38   GEN      sys/time.h
00:00:38   GEN      sys/types.h
[...]
00:00:50   CC       minibuLoading loadup.el (source)...
00:00:50 dump mode: pbootstrap

So the build of the temacs binary just takes 28 seconds.

00:00:50 Loading emacs-lisp/byte-run (source)...
00:00:50 Loading emacs-lisp/backquote (source)...
[...]
00:01:02 Dump complete

And dumping takes 12 seconds.  Then some other bits are compiled and
then we start the pbootstrap:

00:01:07 ./temacs --batch  -l loadup --temacs=pbootstrap

This seems to proceed in parallell until we reach:

00:01:17   INFO     Scraping files for loaddefs.el... 
[...]
00:01:54   INFO     Scraping files for loaddefs.el...done

Which is single-threaded and takes a whopping 37 seconds!  Could that
possibly be parallelised in some way?  Probably be tricky...

00:01:54 Loading loadup.el (source)...
[...]
00:01:58 Dump complete

Another dump; takes just four seconds.

00:02:00   ELC      ../lisp/language/khmer.elc
00:02:00   ELC      ../lisp/language/korean.elc
[...]
00:02:46   ELC      progmodes/cc-styles.elc
00:02:46   ELC      progmodes/cc-mode.elc

And finally, all the remaining (undumped) .el files are compiled, and
this uses all the cores, so it takes just 46 seconds.

Hm...  well, I guess if somebody wanted to speed up compilation, then
the loaddefs scraping is the biggest single item here to look at.

But to sum up, (+ 22 12 37 4) => 75 seconds of the Emacs compilation is
single-threaded, while 90 seconds are multi-threaded (and as machines
get more and more cores, we'll probably see the single-threaded parts
take more than 50% of the time spent).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Compilation speed

[Paul Eggert]

On 8/6/20 8:20 AM, Lars Ingebrigtsen wrote:

> And then ./configure takes 19 seconds, and that's fully single-threaded,
> I believe?  And...  I'm guessing there's no way to get that to be
> multi-threaded?

You'd either have to rework Autoconf to make it multiprocess (difficult) or 
split configure.ac into pieces and run several './configure' pieces in parallel 
(this might be doable but would still be tricky).

> 00:01:17   INFO     Scraping files for loaddefs.el...
> [...]
> 00:01:54   INFO     Scraping files for loaddefs.el...done
> 
> Which is single-threaded and takes a whopping 37 seconds!  Could that
> possibly be parallelised in some way?  Probably be tricky...

Yes. But this is probably the lowest-hanging fruit.

Re: Compilation speed

[Stefan Monnier]

> 00:00:22 config.status: executing etc-refcards-emacsver.tex commands
>
> And then ./configure takes 19 seconds, and that's fully single-threaded,
> I believe?  And...  I'm guessing there's no way to get that to be
> multi-threaded?

This is a long standing and well known issue.
The solutions so far are either:

- move to something else (like `cmake`)
- live with it

IIRC Tom Tromey had started work on some autoconf replacement written in
GMake, but...

I believe the most promising practical solution is to work on
incrementally (i.e. over many years) splitting Autoconf tests into
separate scripts and add a GMake wrapper around them to run them
in parallel.

This is largely orthogonal to Emacs.

> 00:00:50 Loading emacs-lisp/byte-run (source)...
> 00:00:50 Loading emacs-lisp/backquote (source)...
> [...]
> 00:01:02 Dump complete
>
> And dumping takes 12 seconds.

For re-builds, this is also a very significant chunk, especially since
we often do it twice.

> This seems to proceed in parallell until we reach:
>
> 00:01:17   INFO     Scraping files for loaddefs.el... 
> [...]
> 00:01:54   INFO     Scraping files for loaddefs.el...done
>
> Which is single-threaded and takes a whopping 37 seconds!  Could that
> possibly be parallelised in some way?

IIRC this should be generated in parallel with the compilation of the
preloaded files.  Not sure why it's not the case.

> But to sum up, (+ 22 12 37 4) => 75 seconds of the Emacs compilation is
> single-threaded, while 90 seconds are multi-threaded (and as machines
> get more and more cores, we'll probably see the single-threaded parts
> take more than 50% of the time spent).

Note that the native-compiler should solve a significant part of this
problem (by increasing the amount of time spent compiling the .elc files
in parallel ;-)


        Stefan

Re: Compilation speed

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> 00:00:50 Loading emacs-lisp/byte-run (source)...
>> 00:00:50 Loading emacs-lisp/backquote (source)...
>> [...]
>> 00:01:02 Dump complete
>>
>> And dumping takes 12 seconds.
>
> For re-builds, this is also a very significant chunk, especially since
> we often do it twice.

Yes, that's true.

>> This seems to proceed in parallell until we reach:
>>
>> 00:01:17   INFO     Scraping files for loaddefs.el... 
>> [...]
>> 00:01:54   INFO     Scraping files for loaddefs.el...done
>>
>> Which is single-threaded and takes a whopping 37 seconds!  Could that
>> possibly be parallelised in some way?
>
> IIRC this should be generated in parallel with the compilation of the
> preloaded files.  Not sure why it's not the case.

On my laptop (which is much slower), the loaddefs generation often runs
in parallel with other things (well, sort of -- it's usually completed
about a quarter of what it's doing when make runs out of other things to
schedule).

> Note that the native-compiler should solve a significant part of this
> problem (by increasing the amount of time spent compiling the .elc files
> in parallel ;-)

:-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Compilation speed

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:


> Note that the native-compiler should solve a significant part of this
> problem (by increasing the amount of time spent compiling the .elc files
> in parallel ;-)

Doing my best!! :D

  Andrea

-- 
akrl@sdf.org

Re: Compilation speed

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> A nice thing about new, shiny machines with a bunch of cores is that it
> makes saying "make bootstrap" faster when I want to check stuff.
>
> However, there's a couple of sections that are single-threaded, which
> makes the machine mostly idle while building, and I wondered whether
> anybody has looked into these things...
>
> I did a bootstrap (with make -j8) and timestamped the lines to see
> whether anything in particular stands out.
>
> [larsi@stories ~/src/emacs/trunk]$ emake bootstrap 2>&1 | ts -s
> 00:00:00 rm -f libXMenu11.a ./*.o deps/*
> ...
>
> First deleting a lot of stuff...  that's fine...
>
> 00:00:03 running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion
> 00:00:03 checking for xcrun... no
> 00:00:03 checking for GNU Make... make
>
> [...]
>
> 00:00:22 config.status: executing etc-refcards-emacsver.tex commands
>
>
> But I've also long wondered whether anybody has actually pruned the
> tests lately?  Are all those tests (for things that seem really trivial,
> mostly) really necessary these days?

I have wondered this for years when I am compiling/recompiling (mostly
GNU) software. I think configure is great for configuring options, but I
also wonder if it needs to check for all that C compiler portability.
Since Emacs requires C99 compiler (as I understand) couldn't configure
script just check for avialability of C99 standard, OS and compiler and
assume the headers, flags and certain functions are avialable?In essence
it already does this, but also checks for all kind of stuff that nowdays
is standard (like stdlib.h, string.h etc).

First checks on my system are those:

00:00:00 checking for GNU Make... make
00:00:00 checking build system type... x86_64-pc-linux-gnu
00:00:00 checking host system type... x86_64-pc-linux-gnu
00:00:00 checking for gcc... gcc

So if configure already detected GCC does it really need to check for
all those -fno-common and rest of the flags afterwards; couldn't it be
assumed 'yes' to all those automatically? As well that we have stdint.h,
limits.h and OS stuff sys/*.h, malloc, dirent, etc. I am not so
knowledgable about autotools, but if there is for example not strnlen
function avialable, would it be copied/added to system to by autotools
or would it just bail out and says it can't create executable?

For me it takes about 18 seconds to do all those tests, and 1 second to create
makefiles, so it would be cool if configure script could skipp some of
those tests.

> And then ./configure takes 19 seconds, and that's fully single-threaded,
> I believe?  And...  I'm guessing there's no way to get that to be
> multi-threaded?

As I read in autotools, lots of those tests will actually create small
application, compile it, run it and record the result? Or do I
missunderstand? Couldn't autotools devs, make those tests run in their
own shell as asynchronous processes, write output to a file and then
make a finall pass over all outputs and then decide to bail out/continue
or take an action. I guess won't happen, just a rambling.

Re: Compilation speed

[Paul Eggert]

On 8/10/20 7:33 AM, Arthur Miller wrote:

> Since Emacs requires C99 compiler (as I understand) couldn't configure
> script just check for avialability of C99 standard, OS and compiler and
> assume the headers, flags and certain functions are avialable?

Unfortunately not. Lots of implementations conform mostly to C99 but have some 
exceptions. So we can't have a single C99 test; we need individual tests to 
cover the exceptions.

That's not to say we couldn't prune 'configure'; we could. It has several tests 
that are no longer relevant on today's porting targets. The problem is finding 
the developer time to prune them. There's a lot of good stuff there mixed in 
with the cruft.

> if there is for example not strnlen
> function avialable, would it be copied/added to system to by autotools

Generally speaking the autotools don't mess with your dev-chain installation, 
for understandable reasons.

> Couldn't autotools devs, make those tests run in their
> own shell as asynchronous processes, write output to a file and then
> make a finall pass over all outputs and then decide to bail out/continue
> or take an action.

Yes, but this would require quite a rewrite, as existing scripts typically don't 
say which tests depend on which other tests' results. I'm not saying it couldn't 
be done, only that it wouldn't be that easy.

Re: Compilation speed

[Arthur Miller]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 8/10/20 7:33 AM, Arthur Miller wrote:
>
>> Since Emacs requires C99 compiler (as I understand) couldn't configure
>> script just check for avialability of C99 standard, OS and compiler and
>> assume the headers, flags and certain functions are avialable?
>
> Unfortunately not. Lots of implementations conform mostly to C99 but have some
> exceptions. So we can't have a single C99 test; we need individual tests to 
> cover the exceptions.
Ok, I understand; does it mean that different options are chosen per
compiler; or does configure bail out? I am sorry if I ask obvious/stupid
question, I am not so familiar with how autotools works under the hood, despite
executing that ./configure so many times :-).

Can't we say: you need a c99 conformant compiler, otherwise
compile will fail; and for those that use not fully conformant compiler,
let them be on their own? That is how lots of software nowdays work;
they will tell the requirements, and is up to end-user to meet those.

> That's not to say we couldn't prune 'configure'; we could. It has several tests
> that are no longer relevant on today's porting targets. The problem is finding 
> the developer time to prune them. There's a lot of good stuff there mixed in
> with the cruft.

Yes, and trick is to know which are safe to remove.

>> Couldn't autotools devs, make those tests run in their
>> own shell as asynchronous processes, write output to a file and then
>> make a finall pass over all outputs and then decide to bail out/continue
>> or take an action.
>
> Yes, but this would require quite a rewrite, as existing scripts typically don't
> say which tests depend on which other tests' results. I'm not saying it couldn't 
> be done, only that it wouldn't be that easy.

Yes, I understand that :-). Would require new/modern autotools
implementation written for a parallel computer.

Re: Compilation speed

[Stefan Monnier]

> Can't we say: you need a c99 conformant compiler, otherwise

IIUC 100% conformant compilers are pretty hard to come by,


        Stefan

Re: Compilation speed

[Paul Eggert]

On 8/12/20 5:53 AM, Arthur Miller wrote:

> Ok, I understand; does it mean that different options are chosen per
> compiler; or does configure bail out?

'configure' does add some compiler options based on your platform (e.g., '-O2') 
but it also respects the options you choose. It bails out only if nothing seems 
to work.

> Can't we say: you need a c99 conformant compiler, otherwise
> compile will fail; and for those that use not fully conformant compiler,
> let them be on their own?

We do say something like that already, in the sense that we require common C99 
features like declaration-after-statement. However, we don't insist on 100% C99 
conformance since (a) we don't need it and (b) it's pretty rare.

Re: Compilation speed

[Mario Lang]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 8/6/20 8:20 AM, Lars Ingebrigtsen wrote:
>
>> And then ./configure takes 19 seconds, and that's fully single-threaded,
>> I believe?  And...  I'm guessing there's no way to get that to be
>> multi-threaded?
>
> You'd either have to rework Autoconf to make it multiprocess
> (difficult) or split configure.ac into pieces and run several
> './configure' pieces in parallel (this might be doable but would still
> be tricky).

The "speed" of autoconf is probably the reason why its adoption rate is
falling.  I remember insisting on using autoconf for many years, just
because.  Until I figured out CMake 3.x.  I will never write an autoconf
script again, never.

-- 
AR Mario Lang                               Phone: +43 316 873 6897
Graz University of Technology              Mobile: +43 664 60 873 6897
IT-Services for research and teaching       Email: mlang@tugraz.at
Steyrergasse 30/1, 8010 Graz, Austria       Please https://useplaintext.email/

Re: Compilation speed

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

On Wed, Aug 19, 2020 at 11:25:37AM +0200, Mario Lang wrote:
> Paul Eggert <eggert@cs.ucla.edu> writes:
> 
> > On 8/6/20 8:20 AM, Lars Ingebrigtsen wrote:
> >
> >> And then ./configure takes 19 seconds, and that's fully single-threaded,
> >> I believe?  And...  I'm guessing there's no way to get that to be
> >> multi-threaded?

[...]

> The "speed" of autoconf is probably the reason why its adoption rate is
> falling.  I remember insisting on using autoconf for many years, just
> because.  Until I figured out CMake 3.x.  I will never write an autoconf
> script again, never.

I used to consider CMake as "just another build system, why not". Until
I saw its cross-compilation story. Since then, I appreciate autoconf
even more.

Speed (more precisely: parallelizability) isn't that important to me.
Usually I've got other nice things to do while Emacs compiles (i.e.
I schedule such jobs) -- although I do understand that other peoples's
workflows are different.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Compilation speed

[Ergus]

On Wed, Aug 19, 2020 at 11:43:05AM +0200, tomas@tuxteam.de wrote:
>On Wed, Aug 19, 2020 at 11:25:37AM +0200, Mario Lang wrote:
>> Paul Eggert <eggert@cs.ucla.edu> writes:
>>
>> > On 8/6/20 8:20 AM, Lars Ingebrigtsen wrote:
>> >
>> >> And then ./configure takes 19 seconds, and that's fully single-threaded,
>> >> I believe?  And...  I'm guessing there's no way to get that to be
>> >> multi-threaded?
>
>[...]
>
>> The "speed" of autoconf is probably the reason why its adoption rate is
>> falling.  I remember insisting on using autoconf for many years, just
>> because.  Until I figured out CMake 3.x.  I will never write an autoconf
>> script again, never.
>
>I used to consider CMake as "just another build system, why not". Until
>I saw its cross-compilation story. Since then, I appreciate autoconf
>even more.
>
I do cross compilation in CMake almost daily and I actually have very
few complains. It is extremely simple and it has "A syntax to rule them
all". And other "extras" like cpack, ctest, ccmake, the finders (with
the same syntax) or the CMAKE_EXPORT_COMPILE_COMMANDS option to generato
compilation database for clang.

The only real issue for me in CMake is that there is not a log where I
can see quickly the configuration command I used on yesterday.

OTOH, in emacs CMake is not an option because gnulib does not give any
support for it (developers policy). Which in my opinion is bad for
gnulib because CMake use is growing.