please make byte compiling during bootstrap take advantage of make -j

please make byte compiling during bootstrap take advantage of make -j

[Dan Nicolaescu]

Multi-core CPUs are already widespread, and will only become more so.

Byte compiling all the elisp files is where most of the time is spent
during bootstrap.  But currently byte compilation does not take
advantage of "make -j", which would speed it up a great deal.

Can you please implement this?

Thanks

Re: please make byte compiling during bootstrap take advantage of make -j

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@ics.uci.edu>
> Date: Sun, 04 May 2008 10:35:36 -0700
> 
> 
> Multi-core CPUs are already widespread, and will only become more so.
> 
> Byte compiling all the elisp files is where most of the time is spent
> during bootstrap.  But currently byte compilation does not take
> advantage of "make -j", which would speed it up a great deal.
> 
> Can you please implement this?

There's this comment in lisp/Makefile.in:

    #                                      Files MUST be compiled one by
    # one.  If we compile several files in a row we can't make sure that
    # the compilation environment is clean.

I have only a vague idea of what is this trying to say, but assuming
that this consideration is not important for several compilations
going on in parallel, doing what you want boils down to making the
`compile:' target depend on all the .elc files; the rest will be done
automatically by Make.

Re: please make byte compiling during bootstrap take advantage of make -j

[Glenn Morris]

Eli Zaretskii wrote:

> There's this comment in lisp/Makefile.in:
>
>     #                                      Files MUST be compiled one by
>     # one.  If we compile several files in a row we can't make sure that
>     # the compilation environment is clean.
>
> I have only a vague idea of what is this trying to say,

I think this means using:

emacs -f batch-byte-compile-if-not-done only-one-file.el

rather than 

emacs -f batch-byte-compile-if-not-done file1.el file2.el ... fileN.el

In the latter case, stuff defined in file1 might pollute the
compilation environment for file2.

> but assuming that this consideration is not important for several
> compilations going on in parallel,

If the above interpretation is correct, it would not be, and it would
be totally fine to compile in parallel (after those few .el files that
need to be done first).

> doing what you want boils down to making the `compile:' target
> depend on all the .elc files; the rest will be done automatically by
> Make.

Re: please make byte compiling during bootstrap take advantage of make -j

[Glenn Morris]

>> doing what you want boils down to making the `compile:' target
>> depend on all the .elc files; the rest will be done automatically by
>> Make.

But how to do this without hard-coding the names of all the .elc files?

With GNU make, I would try something like:

  compile: $(shell find . -name '*.el' | sed 's/\.el$$/.elc/' )

but I'm sure this is not portable.

Re: please make byte compiling during bootstrap take advantage of make -j

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: bug-gnu-emacs@gnu.org,  Dan Nicolaescu <dann@ics.uci.edu>
> Date: Tue, 06 May 2008 23:05:06 -0400
> 
> 
> >> doing what you want boils down to making the `compile:' target
> >> depend on all the .elc files; the rest will be done automatically by
> >> Make.
> 
> But how to do this without hard-coding the names of all the .elc files?

And what's wrong with hard-coding them, exactly?

If you are worried about maintenance, then we could have a
maintainer-only target that would update the list, and that target
could use GNU Make features, like you suggest below.

> With GNU make, I would try something like:
> 
>   compile: $(shell find . -name '*.el' | sed 's/\.el$$/.elc/' )
> 
> but I'm sure this is not portable.

Actually, you don't need the Sed part, as $(FOO:.el=.elc) is portable.

Re: please make byte compiling during bootstrap take advantage of make -j

[Richard M Stallman]

One idea is to divide the .el files according to the first letters
of their names.  You could divide them into 26 categories this way,
and have 26 targets which could be run in parallel in any manner.
Since each target would compile a substantial number of files,
this would not greatly increase the overhead of starting processes
for compilation.

Re: please make byte compiling during bootstrap take advantage of make -j

[Glenn Morris]

Eli Zaretskii wrote:

> And what's wrong with hard-coding them, exactly?

Just that it's inelegant and more work. Even the directory names
aren't hard-coded at present.

> If you are worried about maintenance, then we could have a
> maintainer-only target that would update the list, and that target
> could use GNU Make features, like you suggest below.

If the names aren't going in a prerequisite, there's no need to use
GNU Makeisms.

So, I envisage:

1. A rule run at the end of bootstrap that finds all the .elc files
and updates Makefile.in (yuck; I'd prefer an include file, but it
seems this is not portable either?).

2. The compile targets changed to use the explicit list of .elc files
constructed above.

3. The compile targets to run something like their current
implementations after the parallelizable part, to catch any new .elc
files not yet added to Makefile.in. Perhaps this isn't necessary,
since all it means is new files wouldn't get compiled until
Makefile.in was updated.

Re: please make byte compiling during bootstrap take advantage of make -j

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: bug-gnu-emacs@gnu.org,  dann@ics.uci.edu
> Date: Wed, 07 May 2008 18:33:41 -0400
> 
> Eli Zaretskii wrote:
> 
> > And what's wrong with hard-coding them, exactly?
> 
> Just that it's inelegant and more work. Even the directory names
> aren't hard-coded at present.

We already have similar lists in src/Makefile.in and in
leim/Makefile.in, so I don't see a problem with having a full list in
lisp/Makefile.in.  We don't add files frequently enough to make that a
bother, IMO.

> So, I envisage:
> 
> 1. A rule run at the end of bootstrap that finds all the .elc files
> and updates Makefile.in (yuck; I'd prefer an include file, but it
> seems this is not portable either?).

Yes, include is non-portable.

But I don't think this will work: you need to know the list of files
to compile in advance, before the bootstrap.  Doing it only after the
bootstrap, and relying on .elc files, would be a chicken-and-egg
problem, no?

I think you can safely have a full list of all .el files, and let the
no-byte-compile tag take care of which ones are actually compiled.

> 2. The compile targets changed to use the explicit list of .elc files
> constructed above.

You could use the list of .el files with .el=.elc construct.

Re: please make byte compiling during bootstrap take advantage of make -j

[Glenn Morris]

Eli Zaretskii wrote:

> Yes, include is non-portable.

Is there a good reference for what is portable? Ideally, for what
Emacs wants to support...

In the GNU Make manual, "Features of GNU make" seems only to say that
including multiple files with a single include is a GNU extension.

And in the section "Old-Fashioned Suffix Rules", it says that "Suffix
rules cannot have any prerequisites of their own", yet we have:

.el.elc: $(lisp)/subdirs.el

in lisp/Makefile.

> I think you can safely have a full list of all .el files, and let the
> no-byte-compile tag take care of which ones are actually compiled.

Yes, that seems to work.

> You could use the list of .el files with .el=.elc construct.

Can I portably use this kind of thing:

lispfiles1 = $(lisp)/*.el
lispfiles2 = $(lisp)/*/*.el
elcfiles1 = $(lispfiles1:.el=.elc)
elcfiles2 = $(lispfiles2:.el=.elc)

compile: $(elcfiles1) $(elcfiles2)

?

Re: please make byte compiling during bootstrap take advantage of make -j

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: bug-gnu-emacs@gnu.org,  dann@ics.uci.edu
> Date: Thu, 08 May 2008 15:34:55 -0400
> 
> Is there a good reference for what is portable? Ideally, for what
> Emacs wants to support...

Not that I know of.

> Can I portably use this kind of thing:
> 
> lispfiles1 = $(lisp)/*.el
> lispfiles2 = $(lisp)/*/*.el
> elcfiles1 = $(lispfiles1:.el=.elc)
> elcfiles2 = $(lispfiles2:.el=.elc)
> 
> compile: $(elcfiles1) $(elcfiles2)
> 
> ?

I'm nor sure.  But I don't consider myself an expert on this; I
suggest to post a question to help-make@gnu.org, there are a few
people there that can run circles around what I know about Make.  In
fact, perhaps it's a good idea to ask them what would be the best way
of portably introducing parallelism into lisp/Makefile.in, they could
come up with some interesting ideas.

Re: please make byte compiling during bootstrap take advantage of make -j

[Dan Nicolaescu]

Glenn Morris <rgm@gnu.org> writes:
  > 
  > Can I portably use this kind of thing:
  > 
  > lispfiles1 = $(lisp)/*.el
  > lispfiles2 = $(lisp)/*/*.el
  > elcfiles1 = $(lispfiles1:.el=.elc)
  > elcfiles2 = $(lispfiles2:.el=.elc)
  > 
  > compile: $(elcfiles1) $(elcfiles2)

IMO if doing things that we need portably is too hard, we should just
switch to requiring GNU make by default.  It is widely available, and
shipped by default on many platforms and it would help simplify and make
easier to maintain a lot of things.

Re: please make byte compiling during bootstrap take advantage of make -j

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@ics.uci.edu>
> Cc: Eli Zaretskii <eliz@gnu.org>, bug-gnu-emacs@gnu.org
> Date: Fri, 09 May 2008 00:02:43 -0700
> 
> Glenn Morris <rgm@gnu.org> writes:
>   > 
>   > Can I portably use this kind of thing:
>   > 
>   > lispfiles1 = $(lisp)/*.el
>   > lispfiles2 = $(lisp)/*/*.el
>   > elcfiles1 = $(lispfiles1:.el=.elc)
>   > elcfiles2 = $(lispfiles2:.el=.elc)
>   > 
>   > compile: $(elcfiles1) $(elcfiles2)
> 
> IMO if doing things that we need portably is too hard, we should just
> switch to requiring GNU make by default.

It's not hard, so we have no good reasons to do that.

bug#189: marked as done (please make byte compiling during bootstrap take advantage of make -j)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]


Your message dated Thu, 29 May 2008 19:10:42 -0400
with message-id <18495.14322.3139.653674@fencepost.gnu.org>
and subject line Re: please make byte compiling during bootstrap take advantage of make -j 
has caused the Emacs bug report #189,
regarding please make byte compiling during bootstrap take advantage of make -j
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact don@donarmstrong.com
immediately.)


-- 
189: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=189
Emacs Bug Tracking System
Contact don@donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 4304 bytes --]

From: Dan Nicolaescu <dann@ics.uci.edu>
To: bug-gnu-emacs@gnu.org
Subject: please make byte compiling during bootstrap take advantage of make -j
Date: Sun, 04 May 2008 10:35:36 -0700
Message-ID: <200805041735.m44HZbxS018196@sallyv1.ics.uci.edu>


Multi-core CPUs are already widespread, and will only become more so.

Byte compiling all the elisp files is where most of the time is spent
during bootstrap.  But currently byte compilation does not take
advantage of "make -j", which would speed it up a great deal.

Can you please implement this?

Thanks





[-- Attachment #3: Type: message/rfc822, Size: 1279 bytes --]

From: Glenn Morris <rgm@gnu.org>
To: 189-done@emacsbugs.donarmstrong.com
Subject: Re: please make byte compiling during bootstrap take advantage of make -j
Date: Thu, 29 May 2008 19:10:42 -0400
Message-ID: <18495.14322.3139.653674@fencepost.gnu.org>


This has been fixed.