* Compilation speed
@ 2020-08-06 15:20 Lars Ingebrigtsen
2020-08-06 19:07 ` Paul Eggert
` (2 more replies)
0 siblings, 3 replies; 13+ messages in thread
From: Lars Ingebrigtsen @ 2020-08-06 15:20 UTC (permalink / raw)
To: emacs-devel
A nice thing about new, shiny machines with a bunch of cores is that it
makes saying "make bootstrap" faster when I want to check stuff.
However, there's a couple of sections that are single-threaded, which
makes the machine mostly idle while building, and I wondered whether
anybody has looked into these things...
I did a bootstrap (with make -j8) and timestamped the lines to see
whether anything in particular stands out.
[larsi@stories ~/src/emacs/trunk]$ emake bootstrap 2>&1 | ts -s
00:00:00 rm -f libXMenu11.a ./*.o deps/*
...
First deleting a lot of stuff... that's fine...
00:00:03 running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion
00:00:03 checking for xcrun... no
00:00:03 checking for GNU Make... make
[...]
00:00:22 config.status: executing etc-refcards-emacsver.tex commands
And then ./configure takes 19 seconds, and that's fully single-threaded,
I believe? And... I'm guessing there's no way to get that to be
multi-threaded?
But I've also long wondered whether anybody has actually pruned the
tests lately? Are all those tests (for things that seem really trivial,
mostly) really necessary these days?
And then compilation of Emacs and the manual starts, and that bit uses
all eight cored:
00:00:22 GEN alloca.h
[...]
00:00:22 GEN ../../info/eintr.info
00:00:22 GEN sys/random.h
00:00:22 GEN sys/select.h
00:00:22 GEN ../../info/elisp.info
00:00:38 GEN sys/stat.h
00:00:38 GEN sys/time.h
00:00:38 GEN sys/types.h
[...]
00:00:50 CC minibuLoading loadup.el (source)...
00:00:50 dump mode: pbootstrap
So the build of the temacs binary just takes 28 seconds.
00:00:50 Loading emacs-lisp/byte-run (source)...
00:00:50 Loading emacs-lisp/backquote (source)...
[...]
00:01:02 Dump complete
And dumping takes 12 seconds. Then some other bits are compiled and
then we start the pbootstrap:
00:01:07 ./temacs --batch -l loadup --temacs=pbootstrap
This seems to proceed in parallell until we reach:
00:01:17 INFO Scraping files for loaddefs.el...
[...]
00:01:54 INFO Scraping files for loaddefs.el...done
Which is single-threaded and takes a whopping 37 seconds! Could that
possibly be parallelised in some way? Probably be tricky...
00:01:54 Loading loadup.el (source)...
[...]
00:01:58 Dump complete
Another dump; takes just four seconds.
00:02:00 ELC ../lisp/language/khmer.elc
00:02:00 ELC ../lisp/language/korean.elc
[...]
00:02:46 ELC progmodes/cc-styles.elc
00:02:46 ELC progmodes/cc-mode.elc
And finally, all the remaining (undumped) .el files are compiled, and
this uses all the cores, so it takes just 46 seconds.
Hm... well, I guess if somebody wanted to speed up compilation, then
the loaddefs scraping is the biggest single item here to look at.
But to sum up, (+ 22 12 37 4) => 75 seconds of the Emacs compilation is
single-threaded, while 90 seconds are multi-threaded (and as machines
get more and more cores, we'll probably see the single-threaded parts
take more than 50% of the time spent).
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-06 15:20 Compilation speed Lars Ingebrigtsen
@ 2020-08-06 19:07 ` Paul Eggert
2020-08-19 9:25 ` Mario Lang
2020-08-06 22:14 ` Stefan Monnier
2020-08-10 14:33 ` Arthur Miller
2 siblings, 1 reply; 13+ messages in thread
From: Paul Eggert @ 2020-08-06 19:07 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: emacs-devel
On 8/6/20 8:20 AM, Lars Ingebrigtsen wrote:
> And then ./configure takes 19 seconds, and that's fully single-threaded,
> I believe? And... I'm guessing there's no way to get that to be
> multi-threaded?
You'd either have to rework Autoconf to make it multiprocess (difficult) or
split configure.ac into pieces and run several './configure' pieces in parallel
(this might be doable but would still be tricky).
> 00:01:17 INFO Scraping files for loaddefs.el...
> [...]
> 00:01:54 INFO Scraping files for loaddefs.el...done
>
> Which is single-threaded and takes a whopping 37 seconds! Could that
> possibly be parallelised in some way? Probably be tricky...
Yes. But this is probably the lowest-hanging fruit.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-06 19:07 ` Paul Eggert
@ 2020-08-19 9:25 ` Mario Lang
2020-08-19 9:43 ` tomas
0 siblings, 1 reply; 13+ messages in thread
From: Mario Lang @ 2020-08-19 9:25 UTC (permalink / raw)
To: emacs-devel
Paul Eggert <eggert@cs.ucla.edu> writes:
> On 8/6/20 8:20 AM, Lars Ingebrigtsen wrote:
>
>> And then ./configure takes 19 seconds, and that's fully single-threaded,
>> I believe? And... I'm guessing there's no way to get that to be
>> multi-threaded?
>
> You'd either have to rework Autoconf to make it multiprocess
> (difficult) or split configure.ac into pieces and run several
> './configure' pieces in parallel (this might be doable but would still
> be tricky).
The "speed" of autoconf is probably the reason why its adoption rate is
falling. I remember insisting on using autoconf for many years, just
because. Until I figured out CMake 3.x. I will never write an autoconf
script again, never.
--
AR Mario Lang Phone: +43 316 873 6897
Graz University of Technology Mobile: +43 664 60 873 6897
IT-Services for research and teaching Email: mlang@tugraz.at
Steyrergasse 30/1, 8010 Graz, Austria Please https://useplaintext.email/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-19 9:25 ` Mario Lang
@ 2020-08-19 9:43 ` tomas
2020-08-19 12:37 ` Ergus
0 siblings, 1 reply; 13+ messages in thread
From: tomas @ 2020-08-19 9:43 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]
On Wed, Aug 19, 2020 at 11:25:37AM +0200, Mario Lang wrote:
> Paul Eggert <eggert@cs.ucla.edu> writes:
>
> > On 8/6/20 8:20 AM, Lars Ingebrigtsen wrote:
> >
> >> And then ./configure takes 19 seconds, and that's fully single-threaded,
> >> I believe? And... I'm guessing there's no way to get that to be
> >> multi-threaded?
[...]
> The "speed" of autoconf is probably the reason why its adoption rate is
> falling. I remember insisting on using autoconf for many years, just
> because. Until I figured out CMake 3.x. I will never write an autoconf
> script again, never.
I used to consider CMake as "just another build system, why not". Until
I saw its cross-compilation story. Since then, I appreciate autoconf
even more.
Speed (more precisely: parallelizability) isn't that important to me.
Usually I've got other nice things to do while Emacs compiles (i.e.
I schedule such jobs) -- although I do understand that other peoples's
workflows are different.
Cheers
- t
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-19 9:43 ` tomas
@ 2020-08-19 12:37 ` Ergus
0 siblings, 0 replies; 13+ messages in thread
From: Ergus @ 2020-08-19 12:37 UTC (permalink / raw)
To: tomas; +Cc: emacs-devel
On Wed, Aug 19, 2020 at 11:43:05AM +0200, tomas@tuxteam.de wrote:
>On Wed, Aug 19, 2020 at 11:25:37AM +0200, Mario Lang wrote:
>> Paul Eggert <eggert@cs.ucla.edu> writes:
>>
>> > On 8/6/20 8:20 AM, Lars Ingebrigtsen wrote:
>> >
>> >> And then ./configure takes 19 seconds, and that's fully single-threaded,
>> >> I believe? And... I'm guessing there's no way to get that to be
>> >> multi-threaded?
>
>[...]
>
>> The "speed" of autoconf is probably the reason why its adoption rate is
>> falling. I remember insisting on using autoconf for many years, just
>> because. Until I figured out CMake 3.x. I will never write an autoconf
>> script again, never.
>
>I used to consider CMake as "just another build system, why not". Until
>I saw its cross-compilation story. Since then, I appreciate autoconf
>even more.
>
I do cross compilation in CMake almost daily and I actually have very
few complains. It is extremely simple and it has "A syntax to rule them
all". And other "extras" like cpack, ctest, ccmake, the finders (with
the same syntax) or the CMAKE_EXPORT_COMPILE_COMMANDS option to generato
compilation database for clang.
The only real issue for me in CMake is that there is not a log where I
can see quickly the configuration command I used on yesterday.
OTOH, in emacs CMake is not an option because gnulib does not give any
support for it (developers policy). Which in my opinion is bad for
gnulib because CMake use is growing.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-06 15:20 Compilation speed Lars Ingebrigtsen
2020-08-06 19:07 ` Paul Eggert
@ 2020-08-06 22:14 ` Stefan Monnier
2020-08-07 6:55 ` Lars Ingebrigtsen
2020-08-07 16:26 ` Andrea Corallo via Emacs development discussions.
2020-08-10 14:33 ` Arthur Miller
2 siblings, 2 replies; 13+ messages in thread
From: Stefan Monnier @ 2020-08-06 22:14 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: emacs-devel
> 00:00:22 config.status: executing etc-refcards-emacsver.tex commands
>
> And then ./configure takes 19 seconds, and that's fully single-threaded,
> I believe? And... I'm guessing there's no way to get that to be
> multi-threaded?
This is a long standing and well known issue.
The solutions so far are either:
- move to something else (like `cmake`)
- live with it
IIRC Tom Tromey had started work on some autoconf replacement written in
GMake, but...
I believe the most promising practical solution is to work on
incrementally (i.e. over many years) splitting Autoconf tests into
separate scripts and add a GMake wrapper around them to run them
in parallel.
This is largely orthogonal to Emacs.
> 00:00:50 Loading emacs-lisp/byte-run (source)...
> 00:00:50 Loading emacs-lisp/backquote (source)...
> [...]
> 00:01:02 Dump complete
>
> And dumping takes 12 seconds.
For re-builds, this is also a very significant chunk, especially since
we often do it twice.
> This seems to proceed in parallell until we reach:
>
> 00:01:17 INFO Scraping files for loaddefs.el...
> [...]
> 00:01:54 INFO Scraping files for loaddefs.el...done
>
> Which is single-threaded and takes a whopping 37 seconds! Could that
> possibly be parallelised in some way?
IIRC this should be generated in parallel with the compilation of the
preloaded files. Not sure why it's not the case.
> But to sum up, (+ 22 12 37 4) => 75 seconds of the Emacs compilation is
> single-threaded, while 90 seconds are multi-threaded (and as machines
> get more and more cores, we'll probably see the single-threaded parts
> take more than 50% of the time spent).
Note that the native-compiler should solve a significant part of this
problem (by increasing the amount of time spent compiling the .elc files
in parallel ;-)
Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-06 22:14 ` Stefan Monnier
@ 2020-08-07 6:55 ` Lars Ingebrigtsen
2020-08-07 16:26 ` Andrea Corallo via Emacs development discussions.
1 sibling, 0 replies; 13+ messages in thread
From: Lars Ingebrigtsen @ 2020-08-07 6:55 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 00:00:50 Loading emacs-lisp/byte-run (source)...
>> 00:00:50 Loading emacs-lisp/backquote (source)...
>> [...]
>> 00:01:02 Dump complete
>>
>> And dumping takes 12 seconds.
>
> For re-builds, this is also a very significant chunk, especially since
> we often do it twice.
Yes, that's true.
>> This seems to proceed in parallell until we reach:
>>
>> 00:01:17 INFO Scraping files for loaddefs.el...
>> [...]
>> 00:01:54 INFO Scraping files for loaddefs.el...done
>>
>> Which is single-threaded and takes a whopping 37 seconds! Could that
>> possibly be parallelised in some way?
>
> IIRC this should be generated in parallel with the compilation of the
> preloaded files. Not sure why it's not the case.
On my laptop (which is much slower), the loaddefs generation often runs
in parallel with other things (well, sort of -- it's usually completed
about a quarter of what it's doing when make runs out of other things to
schedule).
> Note that the native-compiler should solve a significant part of this
> problem (by increasing the amount of time spent compiling the .elc files
> in parallel ;-)
:-)
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-06 22:14 ` Stefan Monnier
2020-08-07 6:55 ` Lars Ingebrigtsen
@ 2020-08-07 16:26 ` Andrea Corallo via Emacs development discussions.
1 sibling, 0 replies; 13+ messages in thread
From: Andrea Corallo via Emacs development discussions. @ 2020-08-07 16:26 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Lars Ingebrigtsen, emacs-devel
Stefan Monnier <monnier@iro.umontreal.ca> writes:
> Note that the native-compiler should solve a significant part of this
> problem (by increasing the amount of time spent compiling the .elc files
> in parallel ;-)
Doing my best!! :D
Andrea
--
akrl@sdf.org
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-06 15:20 Compilation speed Lars Ingebrigtsen
2020-08-06 19:07 ` Paul Eggert
2020-08-06 22:14 ` Stefan Monnier
@ 2020-08-10 14:33 ` Arthur Miller
2020-08-12 2:11 ` Paul Eggert
2 siblings, 1 reply; 13+ messages in thread
From: Arthur Miller @ 2020-08-10 14:33 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: emacs-devel
Lars Ingebrigtsen <larsi@gnus.org> writes:
> A nice thing about new, shiny machines with a bunch of cores is that it
> makes saying "make bootstrap" faster when I want to check stuff.
>
> However, there's a couple of sections that are single-threaded, which
> makes the machine mostly idle while building, and I wondered whether
> anybody has looked into these things...
>
> I did a bootstrap (with make -j8) and timestamped the lines to see
> whether anything in particular stands out.
>
> [larsi@stories ~/src/emacs/trunk]$ emake bootstrap 2>&1 | ts -s
> 00:00:00 rm -f libXMenu11.a ./*.o deps/*
> ...
>
> First deleting a lot of stuff... that's fine...
>
> 00:00:03 running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion
> 00:00:03 checking for xcrun... no
> 00:00:03 checking for GNU Make... make
>
> [...]
>
> 00:00:22 config.status: executing etc-refcards-emacsver.tex commands
>
>
> But I've also long wondered whether anybody has actually pruned the
> tests lately? Are all those tests (for things that seem really trivial,
> mostly) really necessary these days?
I have wondered this for years when I am compiling/recompiling (mostly
GNU) software. I think configure is great for configuring options, but I
also wonder if it needs to check for all that C compiler portability.
Since Emacs requires C99 compiler (as I understand) couldn't configure
script just check for avialability of C99 standard, OS and compiler and
assume the headers, flags and certain functions are avialable?In essence
it already does this, but also checks for all kind of stuff that nowdays
is standard (like stdlib.h, string.h etc).
First checks on my system are those:
00:00:00 checking for GNU Make... make
00:00:00 checking build system type... x86_64-pc-linux-gnu
00:00:00 checking host system type... x86_64-pc-linux-gnu
00:00:00 checking for gcc... gcc
So if configure already detected GCC does it really need to check for
all those -fno-common and rest of the flags afterwards; couldn't it be
assumed 'yes' to all those automatically? As well that we have stdint.h,
limits.h and OS stuff sys/*.h, malloc, dirent, etc. I am not so
knowledgable about autotools, but if there is for example not strnlen
function avialable, would it be copied/added to system to by autotools
or would it just bail out and says it can't create executable?
For me it takes about 18 seconds to do all those tests, and 1 second to create
makefiles, so it would be cool if configure script could skipp some of
those tests.
> And then ./configure takes 19 seconds, and that's fully single-threaded,
> I believe? And... I'm guessing there's no way to get that to be
> multi-threaded?
As I read in autotools, lots of those tests will actually create small
application, compile it, run it and record the result? Or do I
missunderstand? Couldn't autotools devs, make those tests run in their
own shell as asynchronous processes, write output to a file and then
make a finall pass over all outputs and then decide to bail out/continue
or take an action. I guess won't happen, just a rambling.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-10 14:33 ` Arthur Miller
@ 2020-08-12 2:11 ` Paul Eggert
2020-08-12 12:53 ` Arthur Miller
0 siblings, 1 reply; 13+ messages in thread
From: Paul Eggert @ 2020-08-12 2:11 UTC (permalink / raw)
To: Arthur Miller, Lars Ingebrigtsen; +Cc: emacs-devel
On 8/10/20 7:33 AM, Arthur Miller wrote:
> Since Emacs requires C99 compiler (as I understand) couldn't configure
> script just check for avialability of C99 standard, OS and compiler and
> assume the headers, flags and certain functions are avialable?
Unfortunately not. Lots of implementations conform mostly to C99 but have some
exceptions. So we can't have a single C99 test; we need individual tests to
cover the exceptions.
That's not to say we couldn't prune 'configure'; we could. It has several tests
that are no longer relevant on today's porting targets. The problem is finding
the developer time to prune them. There's a lot of good stuff there mixed in
with the cruft.
> if there is for example not strnlen
> function avialable, would it be copied/added to system to by autotools
Generally speaking the autotools don't mess with your dev-chain installation,
for understandable reasons.
> Couldn't autotools devs, make those tests run in their
> own shell as asynchronous processes, write output to a file and then
> make a finall pass over all outputs and then decide to bail out/continue
> or take an action.
Yes, but this would require quite a rewrite, as existing scripts typically don't
say which tests depend on which other tests' results. I'm not saying it couldn't
be done, only that it wouldn't be that easy.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-12 2:11 ` Paul Eggert
@ 2020-08-12 12:53 ` Arthur Miller
2020-08-12 16:08 ` Stefan Monnier
2020-08-12 16:59 ` Paul Eggert
0 siblings, 2 replies; 13+ messages in thread
From: Arthur Miller @ 2020-08-12 12:53 UTC (permalink / raw)
To: Paul Eggert; +Cc: Lars Ingebrigtsen, emacs-devel
Paul Eggert <eggert@cs.ucla.edu> writes:
> On 8/10/20 7:33 AM, Arthur Miller wrote:
>
>> Since Emacs requires C99 compiler (as I understand) couldn't configure
>> script just check for avialability of C99 standard, OS and compiler and
>> assume the headers, flags and certain functions are avialable?
>
> Unfortunately not. Lots of implementations conform mostly to C99 but have some
> exceptions. So we can't have a single C99 test; we need individual tests to
> cover the exceptions.
Ok, I understand; does it mean that different options are chosen per
compiler; or does configure bail out? I am sorry if I ask obvious/stupid
question, I am not so familiar with how autotools works under the hood, despite
executing that ./configure so many times :-).
Can't we say: you need a c99 conformant compiler, otherwise
compile will fail; and for those that use not fully conformant compiler,
let them be on their own? That is how lots of software nowdays work;
they will tell the requirements, and is up to end-user to meet those.
> That's not to say we couldn't prune 'configure'; we could. It has several tests
> that are no longer relevant on today's porting targets. The problem is finding
> the developer time to prune them. There's a lot of good stuff there mixed in
> with the cruft.
Yes, and trick is to know which are safe to remove.
>> Couldn't autotools devs, make those tests run in their
>> own shell as asynchronous processes, write output to a file and then
>> make a finall pass over all outputs and then decide to bail out/continue
>> or take an action.
>
> Yes, but this would require quite a rewrite, as existing scripts typically don't
> say which tests depend on which other tests' results. I'm not saying it couldn't
> be done, only that it wouldn't be that easy.
Yes, I understand that :-). Would require new/modern autotools
implementation written for a parallel computer.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-12 12:53 ` Arthur Miller
@ 2020-08-12 16:08 ` Stefan Monnier
2020-08-12 16:59 ` Paul Eggert
1 sibling, 0 replies; 13+ messages in thread
From: Stefan Monnier @ 2020-08-12 16:08 UTC (permalink / raw)
To: Arthur Miller; +Cc: Lars Ingebrigtsen, Paul Eggert, emacs-devel
> Can't we say: you need a c99 conformant compiler, otherwise
IIUC 100% conformant compilers are pretty hard to come by,
Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Compilation speed
2020-08-12 12:53 ` Arthur Miller
2020-08-12 16:08 ` Stefan Monnier
@ 2020-08-12 16:59 ` Paul Eggert
1 sibling, 0 replies; 13+ messages in thread
From: Paul Eggert @ 2020-08-12 16:59 UTC (permalink / raw)
To: Arthur Miller; +Cc: Lars Ingebrigtsen, emacs-devel
On 8/12/20 5:53 AM, Arthur Miller wrote:
> Ok, I understand; does it mean that different options are chosen per
> compiler; or does configure bail out?
'configure' does add some compiler options based on your platform (e.g., '-O2')
but it also respects the options you choose. It bails out only if nothing seems
to work.
> Can't we say: you need a c99 conformant compiler, otherwise
> compile will fail; and for those that use not fully conformant compiler,
> let them be on their own?
We do say something like that already, in the sense that we require common C99
features like declaration-after-statement. However, we don't insist on 100% C99
conformance since (a) we don't need it and (b) it's pretty rare.
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2020-08-19 12:37 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-08-06 15:20 Compilation speed Lars Ingebrigtsen
2020-08-06 19:07 ` Paul Eggert
2020-08-19 9:25 ` Mario Lang
2020-08-19 9:43 ` tomas
2020-08-19 12:37 ` Ergus
2020-08-06 22:14 ` Stefan Monnier
2020-08-07 6:55 ` Lars Ingebrigtsen
2020-08-07 16:26 ` Andrea Corallo via Emacs development discussions.
2020-08-10 14:33 ` Arthur Miller
2020-08-12 2:11 ` Paul Eggert
2020-08-12 12:53 ` Arthur Miller
2020-08-12 16:08 ` Stefan Monnier
2020-08-12 16:59 ` Paul Eggert
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).