Speeding up the bootstrap build - a quick hack.

Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Emacs.

I sense some unhappiness about emacs-devel occasioned by my branch
scratch/correct-warning-pos, because the build on the branch is a little
slower.

The following may go some way to fixing this slowdown.  Its idea is to
build the lisp directory first by compiling the byte compiler and then
using this to build the native compiler, rather than building the native
compiler directly from interpreted list.

With the amended makefile, my build took 6m 7s.  Before the amendment,
it was around 7m 16s.  This is a saving of around 15%.

This patch is at the stage of being a crude hack, and it has been tried
only on GNU.  It might work on other systems, though.

Enjoy!


diff --git a/lisp/Makefile.in b/lisp/Makefile.in
index 3a72034463..b81337d2f3 100644
--- a/lisp/Makefile.in
+++ b/lisp/Makefile.in
@@ -85,6 +85,14 @@ BYTE_COMPILE_FLAGS =
 # the compiler (so its compilation does not speed up subsequent compilations),
 # it's only placed here so as to speed up generation of the loaddefs.el file.
 
+COMPILE_ZEROTH = \
+	$(lisp)/emacs-lisp/macroexp.elc0 \
+	$(lisp)/emacs-lisp/cconv.elc0 \
+	$(lisp)/emacs-lisp/byte-opt.elc0 \
+	$(lisp)/emacs-lisp/bytecomp.elc0 \
+	$(lisp)/emacs-lisp/comp-cstr.elc0 \
+	$(lisp)/emacs-lisp/comp.elc0
+
 COMPILE_FIRST = \
 	$(lisp)/emacs-lisp/macroexp.elc \
 	$(lisp)/emacs-lisp/cconv.elc    \
@@ -298,22 +306,30 @@ $(THEFILE)n:
 # subdirectories, to make sure require's and load's in the files being
 # compiled find the right files.
 
-.SUFFIXES: .elc .el
+.SUFFIXES: .elc0 .elc .el
 
 # An old-fashioned suffix rule, which, according to the GNU Make manual,
 # cannot have prerequisites.
 ifeq ($(HAVE_NATIVE_COMP),yes)
-.el.elc:
+.el.elc0:
+	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
+	--exec "(setq load-suffixes '(\".elc0\" \".el\"))" \
+	-f batch-byte-compile $<
+	mv $<c $@
+
+%.elc : %.el $(COMPILE_ZEROTH)
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
+	--exec "(setq load-suffixes '(\".elc0\" \".elc\" \".el\"))" \
 	-l comp -f batch-byte+native-compile $<
 else
 .el.elc:
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $<
 endif
 
-.PHONY: compile-first compile-main compile compile-always
+.PHONY: compile-zeroth compile-first compile-main compile compile-always
 
-compile-first: $(COMPILE_FIRST)
+compile-zeroth: $(COMPILE_ZEROTH)
+compile-first: compile-zeroth $(COMPILE_FIRST)
 
 # In 'compile-main' we could directly do
 #    ... | xargs $(MAKE)


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> The following may go some way to fixing this slowdown.  Its idea is to
> build the lisp directory first by compiling the byte compiler and then
> using this to build the native compiler, rather than building the native
> compiler directly from interpreted list.

Thanks, that's pretty much what I suggested we should do (tho I wasn't
too sure *how* to do that ;-)

> This patch is at the stage of being a crude hack, and it has been tried
> only on GNU.  It might work on other systems, though.

It looks pretty good and clean, actually.

I of course have some comments/questions:

> +COMPILE_ZEROTH = \
> +	$(lisp)/emacs-lisp/macroexp.elc0 \
> +	$(lisp)/emacs-lisp/cconv.elc0 \
> +	$(lisp)/emacs-lisp/byte-opt.elc0 \
> +	$(lisp)/emacs-lisp/bytecomp.elc0 \
> +	$(lisp)/emacs-lisp/comp-cstr.elc0 \
> +	$(lisp)/emacs-lisp/comp.elc0
> +
>  COMPILE_FIRST = \
>  	$(lisp)/emacs-lisp/macroexp.elc \
>  	$(lisp)/emacs-lisp/cconv.elc    \

Can't COMPILE_ZEROTH be made directly from COMPILE_FIRST so we don't
need to maintain two different lists?

>  # An old-fashioned suffix rule, which, according to the GNU Make manual,
>  # cannot have prerequisites.
>  ifeq ($(HAVE_NATIVE_COMP),yes)
> -.el.elc:
> +.el.elc0:
> +	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
> +	--exec "(setq load-suffixes '(\".elc0\" \".el\"))" \
> +	-f batch-byte-compile $<
> +	mv $<c $@

The indentation looks wrong here, tho maybe it's a question of taste
(or it's a problem in the TAB/SPC transfer through email).

> +%.elc : %.el $(COMPILE_ZEROTH)
>  	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
> +	--exec "(setq load-suffixes '(\".elc0\" \".elc\" \".el\"))" \
>  	-l comp -f batch-byte+native-compile $<

Shouldn't this list have `.elc` before `.elc0` so we use the
native-compiled compiler when available?

> -.PHONY: compile-first compile-main compile compile-always
> +.PHONY: compile-zeroth compile-first compile-main compile compile-always
>  
> -compile-first: $(COMPILE_FIRST)
> +compile-zeroth: $(COMPILE_ZEROTH)
> +compile-first: compile-zeroth $(COMPILE_FIRST)

Is this necessary, or is it just helpful to debug the Makefile?


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

> With the amended makefile, my build took 6m 7s.  Before the amendment,
> it was around 7m 16s.  This is a saving of around 15%.

That's a most welcome improvement.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Speeding up the bootstrap build - a quick hack.

[Po Lu]

Alan Mackenzie <acm@muc.de> writes:

> The following may go some way to fixing this slowdown.  Its idea is to
> build the lisp directory first by compiling the byte compiler and then
> using this to build the native compiler, rather than building the native
> compiler directly from interpreted list.
>
> With the amended makefile, my build took 6m 7s.  Before the amendment,
> it was around 7m 16s.  This is a saving of around 15%.

I see similar numbers here.  Great, thanks!

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Stefan.

Thanks for the detailled feedback.

On Mon, Jan 17, 2022 at 15:55:26 -0500, Stefan Monnier wrote:
> > The following may go some way to fixing this slowdown.  Its idea is to
> > build the lisp directory first by compiling the byte compiler and then
> > using this to build the native compiler, rather than building the native
> > compiler directly from interpreted list.

> Thanks, that's pretty much what I suggested we should do (tho I wasn't
> too sure *how* to do that ;-)

:-)

> > This patch is at the stage of being a crude hack, and it has been tried
> > only on GNU.  It might work on other systems, though.

> It looks pretty good and clean, actually.

Oh, thanks!

> I of course have some comments/questions:

> > +COMPILE_ZEROTH = \
> > +	$(lisp)/emacs-lisp/macroexp.elc0 \
> > +	$(lisp)/emacs-lisp/cconv.elc0 \
> > +	$(lisp)/emacs-lisp/byte-opt.elc0 \
> > +	$(lisp)/emacs-lisp/bytecomp.elc0 \
> > +	$(lisp)/emacs-lisp/comp-cstr.elc0 \
> > +	$(lisp)/emacs-lisp/comp.elc0
> > +
> >  COMPILE_FIRST = \
> >  	$(lisp)/emacs-lisp/macroexp.elc \
> >  	$(lisp)/emacs-lisp/cconv.elc    \

> Can't COMPILE_ZEROTH be made directly from COMPILE_FIRST so we don't
> need to maintain two different lists?

It can, yes.  There's no reason for them to be different files at the
moment, so I've done this.

> >  # An old-fashioned suffix rule, which, according to the GNU Make manual,
> >  # cannot have prerequisites.
> >  ifeq ($(HAVE_NATIVE_COMP),yes)
> > -.el.elc:
> > +.el.elc0:
> > +	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
> > +	--exec "(setq load-suffixes '(\".elc0\" \".el\"))" \
> > +	-f batch-byte-compile $<
> > +	mv $<c $@

> The indentation looks wrong here, tho maybe it's a question of taste
> (or it's a problem in the TAB/SPC transfer through email).

I think it's just the TAB going to column 8 whatever.

> > +%.elc : %.el $(COMPILE_ZEROTH)
> >  	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
> > +	--exec "(setq load-suffixes '(\".elc0\" \".elc\" \".el\"))" \
> >  	-l comp -f batch-byte+native-compile $<

> Shouldn't this list have `.elc` before `.elc0` so we use the
> native-compiled compiler when available?

I think so, yes.  I've changed this now.

> > -.PHONY: compile-first compile-main compile compile-always
> > +.PHONY: compile-zeroth compile-first compile-main compile compile-always

> > -compile-first: $(COMPILE_FIRST)
> > +compile-zeroth: $(COMPILE_ZEROTH)
> > +compile-first: compile-zeroth $(COMPILE_FIRST)

> Is this necessary, or is it just helpful to debug the Makefile?

I'm not sure.  I'm a little confused, still.

Could I ask you to give the latest version, below, a quick eyeballing,
please?  Maybe I could install this.



diff --git a/lisp/Makefile.in b/lisp/Makefile.in
index 3a72034463..7df58af05e 100644
--- a/lisp/Makefile.in
+++ b/lisp/Makefile.in
@@ -96,6 +96,10 @@ COMPILE_FIRST +=
 endif
 COMPILE_FIRST += $(lisp)/emacs-lisp/autoload.elc
 
+ifeq ($(HAVE_NATIVE_COMP),yes)
+COMPILE_ZEROTH = $(COMPILE_FIRST:.elc=.elc0)
+endif
+
 # Files to compile early in compile-main.  Works around bug#25556.
 MAIN_FIRST = ./emacs-lisp/eieio.el ./emacs-lisp/eieio-base.el \
   ./cedet/semantic/db.el ./emacs-lisp/cconv.el
@@ -298,22 +302,30 @@ $(THEFILE)n:
 # subdirectories, to make sure require's and load's in the files being
 # compiled find the right files.
 
-.SUFFIXES: .elc .el
+.SUFFIXES: .elc0 .elc .el
 
 # An old-fashioned suffix rule, which, according to the GNU Make manual,
 # cannot have prerequisites.
 ifeq ($(HAVE_NATIVE_COMP),yes)
-.el.elc:
+.el.elc0:
+	$(AM_V_ELC0)$(emacs) $(BYTE_COMPILE_FLAGS) \
+	--exec "(setq load-suffixes '(\".elc0\" \".el\"))" \
+	-f batch-byte-compile $<
+	mv $<c $@
+
+%.elc: %.el $(COMPILE_ZEROTH)
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
+	--exec "(setq load-suffixes '(\".elc\" \".elc0\" \".el\"))" \
 	-l comp -f batch-byte+native-compile $<
 else
 .el.elc:
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $<
 endif
 
-.PHONY: compile-first compile-main compile compile-always
+.PHONY: compile-zeroth compile-first compile-main compile compile-always
 
-compile-first: $(COMPILE_FIRST)
+compile-zeroth: $(COMPILE_ZEROTH)
+compile-first: compile-zeroth $(COMPILE_FIRST)
 
 # In 'compile-main' we could directly do
 #    ... | xargs $(MAKE)
@@ -492,7 +504,7 @@ $(CAL_DIR)/hol-loaddefs.el:
 .PHONY: bootstrap-clean distclean maintainer-clean
 
 bootstrap-clean:
-	find $(lisp) -name '*.elc' $(FIND_DELETE)
+	find $(lisp) \( -name '*.elc' -o -name '*.elc0' \) $(FIND_DELETE)
 	rm -f $(AUTOGENEL)
 
 distclean:
diff --git a/src/verbose.mk.in b/src/verbose.mk.in
index e3f5678303..a8081ee55a 100644
--- a/src/verbose.mk.in
+++ b/src/verbose.mk.in
@@ -26,6 +26,7 @@ AM_V_CC =
 AM_V_CXX =
 AM_V_CCLD =
 AM_V_CXXLD =
+AM_V_ELC0 =
 AM_V_ELC =
 AM_V_ELN =
 AM_V_GEN =
@@ -41,13 +42,16 @@ AM_V_CCLD    = @$(info $   CCLD     $@)
 AM_V_CXXLD   = @$(info $   CXXLD    $@)
 ifeq ($(HAVE_NATIVE_COMP),yes)
 ifeq ($(NATIVE_DISABLED),1)
+AM_V_ELC0    = @$(info $   ELC0     $@)
 AM_V_ELC     = @$(info $   ELC      $@)
 AM_V_ELN =
 else
+AM_V_ELC0    = @$(info $   ELC0     $@)
 AM_V_ELC     = @$(info $   ELC+ELN  $@)
 AM_V_ELN     = @$(info $   ELN      $@)
 endif
 else
+AM_V_ELC0 =
 AM_V_ELC     = @$(info $   ELC      $@)
 AM_V_ELN =
 endif


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

>> > -.PHONY: compile-first compile-main compile compile-always
>> > +.PHONY: compile-zeroth compile-first compile-main compile compile-always
>
>> > -compile-first: $(COMPILE_FIRST)
>> > +compile-zeroth: $(COMPILE_ZEROTH)
>> > +compile-first: compile-zeroth $(COMPILE_FIRST)
>
>> Is this necessary, or is it just helpful to debug the Makefile?
>
> I'm not sure.  I'm a little confused, still.

At least I can't see why `compile-first` should need to depend on
`compile-zeroth` since the

    %.elc: %.el $(COMPILE_ZEROTH)

rule should already give the same result.
So I'd suggest you drop this part of the patch and see if that causes
any kind of trouble.

> +ifeq ($(HAVE_NATIVE_COMP),yes)
> +COMPILE_ZEROTH = $(COMPILE_FIRST:.elc=.elc0)
> +endif

I think we can drop the `ifeq` test here.

As for eyeballing: LGTM, thank you.


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Tue, 18 Jan 2022 11:56:38 +0000, Alan Mackenzie <acm@muc.de> said:


    >> >  # An old-fashioned suffix rule, which, according to the GNU Make manual,
    >> >  # cannot have prerequisites.
    >> >  ifeq ($(HAVE_NATIVE_COMP),yes)
    >> > -.el.elc:
    >> > +.el.elc0:
    >> > +	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
    >> > +	--exec "(setq load-suffixes '(\".elc0\" \".el\"))" \
    >> > +	-f batch-byte-compile $<
    >> > +	mv $<c $@

I think this would be far cleaner and less fragile if you used an
order-only prerequisite instead of inventing new suffixes.

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Robert.

On Tue, Jan 18, 2022 at 14:16:33 +0100, Robert Pluim wrote:
> >>>>> On Tue, 18 Jan 2022 11:56:38 +0000, Alan Mackenzie <acm@muc.de> said:


>     >> >  # An old-fashioned suffix rule, which, according to the GNU Make manual,
>     >> >  # cannot have prerequisites.
>     >> >  ifeq ($(HAVE_NATIVE_COMP),yes)
>     >> > -.el.elc:
>     >> > +.el.elc0:
>     >> > +	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
>     >> > +	--exec "(setq load-suffixes '(\".elc0\" \".el\"))" \
>     >> > +	-f batch-byte-compile $<
>     >> > +	mv $<c $@

> I think this would be far cleaner and less fragile if you used an
> order-only prerequisite instead of inventing new suffixes.

Thanks for the reply.

I'm not actually that experienced with make files.  What does "an
orer-only prerequisite" mean?

There was a problem with the current situation where the existence of a
..elc file prevented the native compiler from working.  So I renamed
these early files to .elc0, to prevent them getting in the way.

How might I do this better?

> Robert
> -- 

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> I think this would be far cleaner and less fragile if you used an
> order-only prerequisite instead of inventing new suffixes.

I can't see how that would work.  The current ELC+ELN rules are for
`%.elc`, and here we also need those same `.elc` but built with
another rule.


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Tue, 18 Jan 2022 14:04:35 +0000, Alan Mackenzie <acm@muc.de> said:

    Alan> Hello, Robert.
    Alan> On Tue, Jan 18, 2022 at 14:16:33 +0100, Robert Pluim wrote:
    >> >>>>> On Tue, 18 Jan 2022 11:56:38 +0000, Alan Mackenzie <acm@muc.de> said:


    >> >> >  # An old-fashioned suffix rule, which, according to the GNU Make manual,
    >> >> >  # cannot have prerequisites.
    >> >> >  ifeq ($(HAVE_NATIVE_COMP),yes)
    >> >> > -.el.elc:
    >> >> > +.el.elc0:
    >> >> > +	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
    >> >> > +	--exec "(setq load-suffixes '(\".elc0\" \".el\"))" \
    >> >> > +	-f batch-byte-compile $<
    >> >> > +	mv $<c $@

    >> I think this would be far cleaner and less fragile if you used an
    >> order-only prerequisite instead of inventing new suffixes.

    Alan> Thanks for the reply.

    Alan> I'm not actually that experienced with make files.  What does "an
    Alan> orer-only prerequisite" mean?

It enforces the order in which things are built:

    default: 1.elc 2.elc

    1.elc: | 2.elc

means that 2.elc must be built before 1.elc. You can mix them with
normal prerequisites, so

    1.elc: 3.elc | 2.elc

is valid as well.

    Alan> There was a problem with the current situation where the existence of a
    Alan> ..elc file prevented the native compiler from working.  So I renamed
    Alan> these early files to .elc0, to prevent them getting in the way.

So if you ensured that the native compiler files were built before
that problematic .elc, then things would be ok?

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Eli Zaretskii]

> Date: Mon, 17 Jan 2022 20:26:08 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> diff --git a/lisp/Makefile.in b/lisp/Makefile.in
> index 3a72034463..b81337d2f3 100644
> --- a/lisp/Makefile.in
> +++ b/lisp/Makefile.in
> @@ -85,6 +85,14 @@ BYTE_COMPILE_FLAGS =
>  # the compiler (so its compilation does not speed up subsequent compilations),
>  # it's only placed here so as to speed up generation of the loaddefs.el file.
>  
> +COMPILE_ZEROTH = \
> +	$(lisp)/emacs-lisp/macroexp.elc0 \
> +	$(lisp)/emacs-lisp/cconv.elc0 \
> +	$(lisp)/emacs-lisp/byte-opt.elc0 \
> +	$(lisp)/emacs-lisp/bytecomp.elc0 \
> +	$(lisp)/emacs-lisp/comp-cstr.elc0 \
> +	$(lisp)/emacs-lisp/comp.elc0
> +
>  COMPILE_FIRST = \
>  	$(lisp)/emacs-lisp/macroexp.elc \
>  	$(lisp)/emacs-lisp/cconv.elc    \
> @@ -298,22 +306,30 @@ $(THEFILE)n:
>  # subdirectories, to make sure require's and load's in the files being
>  # compiled find the right files.
>  
> -.SUFFIXES: .elc .el
> +.SUFFIXES: .elc0 .elc .el
>  
>  # An old-fashioned suffix rule, which, according to the GNU Make manual,
>  # cannot have prerequisites.
>  ifeq ($(HAVE_NATIVE_COMP),yes)
> -.el.elc:
> +.el.elc0:
> +	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
> +	--exec "(setq load-suffixes '(\".elc0\" \".el\"))" \
> +	-f batch-byte-compile $<
> +	mv $<c $@
> +
> +%.elc : %.el $(COMPILE_ZEROTH)
>  	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
> +	--exec "(setq load-suffixes '(\".elc0\" \".elc\" \".el\"))" \
>  	-l comp -f batch-byte+native-compile $<
>  else
>  .el.elc:
>  	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $<
>  endif

Is this .elc0 trick just to avoid the ELC+ELN compilation of
COMPILE_FIRST, and instead first compile them only to .elc and then
compile again to .elc + .eln?

If so, why not use no-native-compile to disable the ELN part?  Since
compile-first is called from src/Makefile, as part of building
bootstrap-emacs, you can do that in the commands there.

If you tried that and it didn't work, can you tell me what I missed?

Thanks.

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Tue, 18 Jan 2022 09:05:08 -0500, Stefan Monnier <monnier@iro.umontreal.ca> said:

    >> I think this would be far cleaner and less fragile if you used an
    >> order-only prerequisite instead of inventing new suffixes.

    Stefan> I can't see how that would work.  The current ELC+ELN rules are for
    Stefan> `%.elc`, and here we also need those same `.elc` but built with
    Stefan> another rule.

Now Iʼm confused. We need to build the .elc files twice?

(BTW, if the use of suffix rules is holding us back, I thought we now
required GNU Make, so we could switch to pattern rules).

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> So if you ensured that the native compiler files were built before
> that problematic .elc, then things would be ok?

The problem is that you need one rule

    bytecomp.elc: bytecomp.el
        ELC ...

to build the byte-compiled compiler, and then another rule:

    bytecomp.elc: bytecomp.el bytecomp.elc comp.elc macroexp.elc cconv.elc ...
        ELC+ELN ...

to build the native compiled compiler using the byte-compiled compiler
(which is much faster than the interpreted compiler).

In an ideal world the second rule would not have `bytecomp.elc` as its
target but would have something like `bytecomp.eln` instead, but we have
not yet been able to teach Make how to compute the name of the generated
`.eln` file (it's not just `bytecomp.eln` but includes some hash of the
Emacs binary).


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Tue, 18 Jan 2022 09:24:04 -0500, Stefan Monnier <monnier@iro.umontreal.ca> said:

    >> So if you ensured that the native compiler files were built before
    >> that problematic .elc, then things would be ok?

    Stefan> The problem is that you need one rule

    Stefan>     bytecomp.elc: bytecomp.el
    Stefan>         ELC ...

    Stefan> to build the byte-compiled compiler, and then another rule:

    Stefan>     bytecomp.elc: bytecomp.el bytecomp.elc comp.elc macroexp.elc cconv.elc ...
    Stefan>         ELC+ELN ...

    Stefan> to build the native compiled compiler using the byte-compiled compiler
    Stefan> (which is much faster than the interpreted compiler).

Ah, weʼre lying to make. No wonder weʼre having problems :-)

    Stefan> In an ideal world the second rule would not have `bytecomp.elc` as its
    Stefan> target but would have something like `bytecomp.eln` instead, but we have
    Stefan> not yet been able to teach Make how to compute the name of the generated
    Stefan> `.eln` file (it's not just `bytecomp.eln` but includes some hash of the
    Stefan> Emacs binary).

Thereʼs no other file or directory name that contains that hash? Could
we do a dummy compile of an empty .el using the native compiler and
derive the hash from that? (and then compile the native compiler with
the byte compiler).

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Tue, 18 Jan 2022 15:35:42 +0100, Robert Pluim <rpluim@gmail.com> said:
    Stefan> In an ideal world the second rule would not have `bytecomp.elc` as its
    Stefan> target but would have something like `bytecomp.eln` instead, but we have
    Stefan> not yet been able to teach Make how to compute the name of the generated
    Stefan> `.eln` file (it's not just `bytecomp.eln` but includes some hash of the
    Stefan> Emacs binary).

    Robert> Thereʼs no other file or directory name that contains that hash? Could
    Robert> we do a dummy compile of an empty .el using the native compiler and
    Robert> derive the hash from that? (and then compile the native compiler with
    Robert> the byte compiler).

Answering my own question:

src/emacs -batch --eval '(message "%s" (comp-el-to-eln-filename \
"./lisp/emacs-lisp/bytecomp.el" (car (last \
native-comp-eln-load-path))))' 2>&1

seems to give the right answer.

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Andrea Corallo]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 18 Jan 2022 09:24:04 -0500, Stefan Monnier <monnier@iro.umontreal.ca> said:
>
>     >> So if you ensured that the native compiler files were built before
>     >> that problematic .elc, then things would be ok?
>
>     Stefan> The problem is that you need one rule
>
>     Stefan>     bytecomp.elc: bytecomp.el
>     Stefan>         ELC ...
>
>     Stefan> to build the byte-compiled compiler, and then another rule:
>
>     Stefan>     bytecomp.elc: bytecomp.el bytecomp.elc comp.elc macroexp.elc cconv.elc ...
>     Stefan>         ELC+ELN ...
>
>     Stefan> to build the native compiled compiler using the byte-compiled compiler
>     Stefan> (which is much faster than the interpreted compiler).
>
> Ah, weʼre lying to make. No wonder weʼre having problems :-)
>
>     Stefan> In an ideal world the second rule would not have `bytecomp.elc` as its
>     Stefan> target but would have something like `bytecomp.eln` instead, but we have
>     Stefan> not yet been able to teach Make how to compute the name of the generated
>     Stefan> `.eln` file (it's not just `bytecomp.eln` but includes some hash of the
>     Stefan> Emacs binary).
>
> Thereʼs no other file or directory name that contains that hash? Could
> we do a dummy compile of an empty .el using the native compiler and
> derive the hash from that? (and then compile the native compiler with
> the byte compiler).
>
> Robert

Part of the hash is computed also using the content of the .el file so
the empty .el trick would not work.

  Andrea

Re: Speeding up the bootstrap build - a quick hack.

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Tue, 18 Jan 2022 16:13:23 +0100
> Cc: Alan Mackenzie <acm@muc.de>, emacs-devel@gnu.org
> 
> >>>>> On Tue, 18 Jan 2022 15:35:42 +0100, Robert Pluim <rpluim@gmail.com> said:
>     Stefan> In an ideal world the second rule would not have `bytecomp.elc` as its
>     Stefan> target but would have something like `bytecomp.eln` instead, but we have
>     Stefan> not yet been able to teach Make how to compute the name of the generated
>     Stefan> `.eln` file (it's not just `bytecomp.eln` but includes some hash of the
>     Stefan> Emacs binary).
> 
>     Robert> Thereʼs no other file or directory name that contains that hash? Could
>     Robert> we do a dummy compile of an empty .el using the native compiler and
>     Robert> derive the hash from that? (and then compile the native compiler with
>     Robert> the byte compiler).
> 
> Answering my own question:
> 
> src/emacs -batch --eval '(message "%s" (comp-el-to-eln-filename \
> "./lisp/emacs-lisp/bytecomp.el" (car (last \
> native-comp-eln-load-path))))' 2>&1
> 
> seems to give the right answer.

Yes, and that needs the working Emacs binary, so it doesn't come for
free.  On top of that, some of the COMPILE_FIRST targets end up in
that directory, others it its preloaded/ subdirectory: something else
that only Emacs knows about.

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> Thereʼs no other file or directory name that contains that hash? Could
> we do a dummy compile of an empty .el using the native compiler and
> derive the hash from that? (and then compile the native compiler with
> the byte compiler).

AFAIK, currently the file name is computed by a function in `comp.el`,
so without further hacking, it requires a compiled src/temacs and it
requires loading `comp.el` (which itself requires loading
a whole bunch of other files) which takes a fairly long time in and of
itself when they're not byte-compiled yet.

Maybe there's another solution: generate both the real .eln file with
the funny hash-complete name and a hardlink/symlink/copy at an
easily-predictable place.  Then Make can use that easily-predictable name
to keep track of dependencies.

But in any case, until we've sorted that out, Alan's approach seems
sane enough and brings a welcome speed improvement (especially welcome
when working on the bootstrap itself).


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> Is this .elc0 trick just to avoid the ELC+ELN compilation of
> COMPILE_FIRST, and instead first compile them only to .elc and then
> compile again to .elc + .eln?

Yes.

> If so, why not use no-native-compile to disable the ELN part?  Since
> compile-first is called from src/Makefile, as part of building
> bootstrap-emacs, you can do that in the commands there.

But we also want to native-compile those files (after we've
byte-compiled them), so we do need two different targets.
Those should ideally be `.elc` first and `.eln` later, but we currently
don't know how to make that work, so Alan suggests to use `.elc0` first
and `.elc` later.


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  emacs-devel@gnu.org
> Date: Tue, 18 Jan 2022 13:40:19 -0500
> 
> > Is this .elc0 trick just to avoid the ELC+ELN compilation of
> > COMPILE_FIRST, and instead first compile them only to .elc and then
> > compile again to .elc + .eln?
> 
> Yes.
> 
> > If so, why not use no-native-compile to disable the ELN part?  Since
> > compile-first is called from src/Makefile, as part of building
> > bootstrap-emacs, you can do that in the commands there.
> 
> But we also want to native-compile those files (after we've
> byte-compiled them), so we do need two different targets.

No, we need two consecutive shell commands under the same target: one
with no-native-compile set, the other without it.

This is the current recipe:

  ifeq ($(DUMPING),pdumper)
  $(bootstrap_pdmp): bootstrap-emacs$(EXEEXT)
	  rm -f $@
	  $(RUN_TEMACS) --batch $(BUILD_DETAILS) -l loadup --temacs=pbootstrap \
		  --bin-dest $(BIN_DESTDIR) --eln-dest $(ELN_DESTDIR)
	  @: Compile some files earlier to speed up further compilation.
	  $(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"
  endif

What I had in mind is to run the last "$(MAKE) -C ../lisp compile-first"
line so that it binds no-native-compile to non-nil, and then is to run
it again without binding that variable, after touch'ing the corresponding
*.el files to force the recompile.

> Those should ideally be `.elc` first and `.eln` later, but we currently
> don't know how to make that work, so Alan suggests to use `.elc0` first
> and `.elc` later.

I know.

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Jan 18, 2022 at 08:14:06 -0500, Stefan Monnier wrote:
> >> > -.PHONY: compile-first compile-main compile compile-always
> >> > +.PHONY: compile-zeroth compile-first compile-main compile compile-always

> >> > -compile-first: $(COMPILE_FIRST)
> >> > +compile-zeroth: $(COMPILE_ZEROTH)
> >> > +compile-first: compile-zeroth $(COMPILE_FIRST)

> >> Is this necessary, or is it just helpful to debug the Makefile?

> > I'm not sure.  I'm a little confused, still.

It seems to be necessary.  At any rate, changing the mix slightly gave
rise to unwanted results.  In particular...

> At least I can't see why `compile-first` should need to depend on
> `compile-zeroth` since the

>     %.elc: %.el $(COMPILE_ZEROTH)

If I change that line to

    %.elc: %.el compile-zeroth

, then Emacs builds, but redundantly ELC's all the .el files which are
preloaded, taking 15 seconds longer to do so.  I don't understand why
this happens.

Even more notably, if I eliminate compile-zeroth, putting in instead
$(COMPILE_ZEROTH) everywhere needed, then make creates and deletes the
..elc0 files four times, and also redundantly runs ELC on the preloaded
..el files, despite them already being .eln's.  This run took 2½ minutes
longer than expected, too.  I don't understand why all that happened,
either.

> rule should already give the same result.
> So I'd suggest you drop this part of the patch and see if that causes
> any kind of trouble.

As above, it caused all sorts of trouble.

> > +ifeq ($(HAVE_NATIVE_COMP),yes)
> > +COMPILE_ZEROTH = $(COMPILE_FIRST:.elc=.elc0)
> > +endif

> I think we can drop the `ifeq` test here.

I'll need to check the file will work with native compilation disabled.

> As for eyeballing: LGTM, thank you.

Thanks!

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> What I had in mind is to run the last "$(MAKE) -C ../lisp compile-first"
> line so that it binds no-native-compile to non-nil, and then is to run
> it again without binding that variable, after touch'ing the corresponding
> *.el files to force the recompile.

That could work as well, but `touch`ing the `.el` files is an ugly hack
that's also annoying when you're editing those files (Emacs then asks
you whether you really want to overwrite them, etc...).

I guess we could `touch` the .elc instead (to mark them as older), tho.

I don't personally care which hack we use, and don't find one to be
noticeably cleaner than the other.


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Eli.

On Tue, Jan 18, 2022 at 21:34:38 +0200, Eli Zaretskii wrote:
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: Alan Mackenzie <acm@muc.de>,  emacs-devel@gnu.org
> > Date: Tue, 18 Jan 2022 13:40:19 -0500

> > > Is this .elc0 trick just to avoid the ELC+ELN compilation of
> > > COMPILE_FIRST, and instead first compile them only to .elc and then
> > > compile again to .elc + .eln?

> > Yes.

> > > If so, why not use no-native-compile to disable the ELN part?  Since
> > > compile-first is called from src/Makefile, as part of building
> > > bootstrap-emacs, you can do that in the commands there.

> > But we also want to native-compile those files (after we've
> > byte-compiled them), so we do need two different targets.

> No, we need two consecutive shell commands under the same target: one
> with no-native-compile set, the other without it.

> This is the current recipe:

>   ifeq ($(DUMPING),pdumper)
>   $(bootstrap_pdmp): bootstrap-emacs$(EXEEXT)
> 	  rm -f $@
> 	  $(RUN_TEMACS) --batch $(BUILD_DETAILS) -l loadup --temacs=pbootstrap \
> 		  --bin-dest $(BIN_DESTDIR) --eln-dest $(ELN_DESTDIR)
> 	  @: Compile some files earlier to speed up further compilation.
> 	  $(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"
>   endif

> What I had in mind is to run the last "$(MAKE) -C ../lisp compile-first"
> line so that it binds no-native-compile to non-nil, and then is to run
> it again without binding that variable, after touch'ing the corresponding
> *.el files to force the recompile.

> > Those should ideally be `.elc` first and `.eln` later, but we currently
> > don't know how to make that work, so Alan suggests to use `.elc0` first
> > and `.elc` later.

> I know.

This little project started off life quite simple, but has turned
unbelievably complicated in the last few hours.

I'll have a look at what you're written above tomorrow, in the hope that
it will simplify the recipe I've got at the moment.

But the main thing is that it _is_ possible to reduce how long a native
compilation build takes.  Reducing the compilation time of comp.el from
2min 30sec to 1min 20sec I think is worthwhile.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

>> >> > -.PHONY: compile-first compile-main compile compile-always
>> >> > +.PHONY: compile-zeroth compile-first compile-main compile compile-always
>
>> >> > -compile-first: $(COMPILE_FIRST)
>> >> > +compile-zeroth: $(COMPILE_ZEROTH)
>> >> > +compile-first: compile-zeroth $(COMPILE_FIRST)
>
>> >> Is this necessary, or is it just helpful to debug the Makefile?
>
>> > I'm not sure.  I'm a little confused, still.
>
> It seems to be necessary.  At any rate, changing the mix slightly gave
> rise to unwanted results.  In particular...
>
>> At least I can't see why `compile-first` should need to depend on
>> `compile-zeroth` since the
>
>>     %.elc: %.el $(COMPILE_ZEROTH)
>
> If I change that line to
>
>     %.elc: %.el compile-zeroth
>
> , then Emacs builds, but redundantly ELC's all the .el files which are
> preloaded, taking 15 seconds longer to do so.  I don't understand why
> this happens.
>
> Even more notably, if I eliminate compile-zeroth, putting in instead
> $(COMPILE_ZEROTH) everywhere needed, then make creates and deletes the
> ..elc0 files four times, and also redundantly runs ELC on the preloaded
> ..el files, despite them already being .eln's.  This run took 2½ minutes
> longer than expected, too.  I don't understand why all that happened,
> either.

But what if you don't define `compile-zeroth`, and you keep:

    compile-first: $(COMPILE_FIRST)

and

    %.elc: %.el $(COMPILE_ZEROTH)

?

>> rule should already give the same result.  So I'd suggest you drop
>> this part of the patch and see if that causes any kind of trouble.
> As above, it caused all sorts of trouble.

Maybe I'm confused but IIUC none of what you tried corresponds to just
not using the above hunk.


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> But the main thing is that it _is_ possible to reduce how long a native
> compilation build takes.  Reducing the compilation time of comp.el from
> 2min 30sec to 1min 20sec I think is worthwhile.

It takes substantially more than 2m30s on my machine, so yes, it's
very worthwhile, thanks.


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Eli.

On Tue, Jan 18, 2022 at 21:34:38 +0200, Eli Zaretskii wrote:
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: Alan Mackenzie <acm@muc.de>,  emacs-devel@gnu.org
> > Date: Tue, 18 Jan 2022 13:40:19 -0500

> > > Is this .elc0 trick just to avoid the ELC+ELN compilation of
> > > COMPILE_FIRST, and instead first compile them only to .elc and then
> > > compile again to .elc + .eln?

> > Yes.

> > > If so, why not use no-native-compile to disable the ELN part?  Since
> > > compile-first is called from src/Makefile, as part of building
> > > bootstrap-emacs, you can do that in the commands there.

> > But we also want to native-compile those files (after we've
> > byte-compiled them), so we do need two different targets.

> No, we need two consecutive shell commands under the same target: one
> with no-native-compile set, the other without it.

No, this would not work.  It is essential to have all seven compile-first
files byte compiled before we start native compiling any of them.  That
is what halves the time taken for the compile-to-native of comp.el.

I don't think we can avoid two separate targets for each of these source
files.

> This is the current recipe:

>   ifeq ($(DUMPING),pdumper)
>   $(bootstrap_pdmp): bootstrap-emacs$(EXEEXT)
> 	  rm -f $@
> 	  $(RUN_TEMACS) --batch $(BUILD_DETAILS) -l loadup --temacs=pbootstrap \
> 		  --bin-dest $(BIN_DESTDIR) --eln-dest $(ELN_DESTDIR)
> 	  @: Compile some files earlier to speed up further compilation.
> 	  $(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"
>   endif

> What I had in mind is to run the last "$(MAKE) -C ../lisp compile-first"
> line so that it binds no-native-compile to non-nil, and then is to run
> it again without binding that variable, after touch'ing the corresponding
> *.el files to force the recompile.

> > Those should ideally be `.elc` first and `.eln` later, but we currently
> > don't know how to make that work, so Alan suggests to use `.elc0` first
> > and `.elc` later.

> I know.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Eli Zaretskii]

> Date: Wed, 19 Jan 2022 11:10:32 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > No, we need two consecutive shell commands under the same target: one
> > with no-native-compile set, the other without it.
> 
> No, this would not work.  It is essential to have all seven compile-first
> files byte compiled before we start native compiling any of them.  That
> is what halves the time taken for the compile-to-native of comp.el.

The following single command in src/Makefile.in

	$(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"

compiles all of the seven dwarfs in one go, so I'm not sure what you
mean by "would not work".  Please elaborate.

> I don't think we can avoid two separate targets for each of these source
> files.

I don't think you understood my proposal, because this reason makes no
sense to me.

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Jan 18, 2022 at 15:48:23 -0500, Stefan Monnier wrote:
> >> >> > -.PHONY: compile-first compile-main compile compile-always
> >> >> > +.PHONY: compile-zeroth compile-first compile-main compile compile-always

> >> >> > -compile-first: $(COMPILE_FIRST)
> >> >> > +compile-zeroth: $(COMPILE_ZEROTH)
> >> >> > +compile-first: compile-zeroth $(COMPILE_FIRST)

> >> >> Is this necessary, or is it just helpful to debug the Makefile?

> >> > I'm not sure.  I'm a little confused, still.

> > It seems to be necessary.  At any rate, changing the mix slightly gave
> > rise to unwanted results.  In particular...

> >> At least I can't see why `compile-first` should need to depend on
> >> `compile-zeroth` since the

> >>     %.elc: %.el $(COMPILE_ZEROTH)

> > If I change that line to

> >     %.elc: %.el compile-zeroth

> > , then Emacs builds, but redundantly ELC's all the .el files which are
> > preloaded, taking 15 seconds longer to do so.  I don't understand why
> > this happens.

> > Even more notably, if I eliminate compile-zeroth, putting in instead
> > $(COMPILE_ZEROTH) everywhere needed, then make creates and deletes the
> > ..elc0 files four times, and also redundantly runs ELC on the preloaded
> > ..el files, despite them already being .eln's.  This run took 2½ minutes
> > longer than expected, too.  I don't understand why all that happened,
> > either.

> But what if you don't define `compile-zeroth`, and you keep:

>     compile-first: $(COMPILE_FIRST)

> and

>     %.elc: %.el $(COMPILE_ZEROTH)

> ?

I think that is what I have just tried again.  The results are as in my
paragraph above beginning "Even more notably, ....".

I don't understand at all what's happening.  Maybe it's something to do
with compile-zeroth being declared .PHONY.  It also might have to do with
make somehow regarding *.elc0 as "intermediate files", and is thus "safe"
to delete them (four times deleted in total).

> >> rule should already give the same result.  So I'd suggest you drop
> >> this part of the patch and see if that causes any kind of trouble.
> > As above, it caused all sorts of trouble.

> Maybe I'm confused but IIUC none of what you tried corresponds to just
> not using the above hunk.

I think I'm confused.  make isn't a simple program.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> I don't understand at all what's happening.  Maybe it's something to do
> with compile-zeroth being declared .PHONY.  It also might have to do with
> make somehow regarding *.elc0 as "intermediate files", and is thus "safe"
> to delete them (four times deleted in total).

Maybe it's related to the following part of Make's info:

    Sometimes a file can be made by a sequence of implicit rules.  For
    example, a file 'N.o' could be made from 'N.y' by running first Yacc and
    then 'cc'.  Such a sequence is called a "chain".
    
    [...]
    
       The second difference is that if 'make' _does_ create B in order to
    update something else, it deletes B later on after it is no longer
    needed.  Therefore, an intermediate file which did not exist before
    'make' also does not exist after 'make'.  'make' reports the deletion to
    you by printing a 'rm -f' command showing which file it is deleting.

In that case, I think we can address this with

    .PRECIOUS: $(COMPILE_ZEROTH)


-- Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

> In that case, I think we can address this with
>
>     .PRECIOUS: $(COMPILE_ZEROTH)

And if I read that page correctly it can also be:

    .PRECIOUS: %.elc0

BTW, you said:
> I think I'm confused.  make isn't a simple program.

I wish there was a simpler replacement for it,
more programmable, less magic, with sane quoting rules.


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Eli.

On Wed, Jan 19, 2022 at 13:46:44 +0200, Eli Zaretskii wrote:
> > Date: Wed, 19 Jan 2022 11:10:32 +0000
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > No, we need two consecutive shell commands under the same target: one
> > > with no-native-compile set, the other without it.

> > No, this would not work.  It is essential to have all seven compile-first
> > files byte compiled before we start native compiling any of them.  That
> > is what halves the time taken for the compile-to-native of comp.el.

> The following single command in src/Makefile.in

> 	$(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"

> compiles all of the seven dwarfs in one go, so I'm not sure what you
> mean by "would not work".  Please elaborate.

> > I don't think we can avoid two separate targets for each of these source
> > files.

> I don't think you understood my proposal, because this reason makes no
> sense to me.

You're right, I didn't.  Maybe I understand it now.

The idea is to leave lisp/Makefile.in unchanged, and make all the
alterations in src/Makefile.in.

The segment of code you cite above builds all of compile-first in one
invocation, so there's no need to worry about mixtures of interpreted
source and .elc.

So, we duplicate that bit of code, setting the Emacs variable
no-native-compile on the command line of the first occurrence.  This
will cause the byte compilation of all of compile-first.

After this bit of new code, we use 'touch' to set the date of these new
*.elc's back to the distant past.  This will ensure that these *.el's
get built again in the next step.

The second duplicate of the old bit of make code will build the .eln's,
using the ("very old") .elc's which are still available inside Emacs.
It should do this reasonably quickly, because it is using *.elc's.

Yes, I agree that this approach is more elegant and surely easier to
maintain than my original hack, if it can be made to work (which it
surely can).  I will look at this this evening.  Thanks for the
suggestion.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Eli Zaretskii]

> Date: Wed, 19 Jan 2022 16:50:11 +0000
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> The idea is to leave lisp/Makefile.in unchanged, and make all the
> alterations in src/Makefile.in.
> 
> The segment of code you cite above builds all of compile-first in one
> invocation, so there's no need to worry about mixtures of interpreted
> source and .elc.
> 
> So, we duplicate that bit of code, setting the Emacs variable
> no-native-compile on the command line of the first occurrence.  This
> will cause the byte compilation of all of compile-first.
> 
> After this bit of new code, we use 'touch' to set the date of these new
> *.elc's back to the distant past.  This will ensure that these *.el's
> get built again in the next step.
> 
> The second duplicate of the old bit of make code will build the .eln's,
> using the ("very old") .elc's which are still available inside Emacs.
> It should do this reasonably quickly, because it is using *.elc's.

Yes, that's the idea.

> Yes, I agree that this approach is more elegant and surely easier to
> maintain than my original hack, if it can be made to work (which it
> surely can).  I will look at this this evening.  Thanks for the
> suggestion.

Thank you.

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Eli.

On Wed, Jan 19, 2022 at 19:03:46 +0200, Eli Zaretskii wrote:
> > Date: Wed, 19 Jan 2022 16:50:11 +0000
> > Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > The idea is to leave lisp/Makefile.in unchanged, and make all the
> > alterations in src/Makefile.in.

> > The segment of code you cite above builds all of compile-first in one
> > invocation, so there's no need to worry about mixtures of interpreted
> > source and .elc.

> > So, we duplicate that bit of code, setting the Emacs variable
> > no-native-compile on the command line of the first occurrence.  This
> > will cause the byte compilation of all of compile-first.

> > After this bit of new code, we use 'touch' to set the date of these new
> > *.elc's back to the distant past.  This will ensure that these *.el's
> > get built again in the next step.

> > The second duplicate of the old bit of make code will build the .eln's,
> > using the ("very old") .elc's which are still available inside Emacs.
> > It should do this reasonably quickly, because it is using *.elc's.

> Yes, that's the idea.

> > Yes, I agree that this approach is more elegant and surely easier to
> > maintain than my original hack, if it can be made to work (which it
> > surely can).  I will look at this this evening.  Thanks for the
> > suggestion.

> Thank you.

OK, things didn't go quite according to plan.  The name compile-first
only really has meaning in lisp/Makefile.in, so most of the change had to
happen there.

What gave trouble was the setting of BYTE_COMPILE_FLAGS.  It contains
(setq load-prefer-newer t), which prevents the new mechanism from
working; the newly build seven .elc files have been given a date in 1970,
so the source files are newer and thus get loaded in place of the .elc's.

This was surprisingly difficult to solve.  There appears to be no way in
make to set a variable depending on what the target is.  The make manual
doesn't say this explicitly, it just depends on vagueness.  After an hour
of searching for such a feature, it gradually dawns on you that there is
no such feature, even though one might be expected.  I'm glad the Emacs
manuals aren't like that.

So I "solved" this problem by passing an argument as an environment
variable from src/Makefile.in to lisp/Makefile.in.  Yuck!  If anybody can
come up with a better way, it would be appreciated.

Anyhow, this new scheme appears to work, and it's almost certainly better
than the first approach.  Here's the current patch.  Comments and
criticisms would be welcome.

Thanks!



diff --git a/lisp/Makefile.in b/lisp/Makefile.in
index 3a72034463..8f4001cf24 100644
--- a/lisp/Makefile.in
+++ b/lisp/Makefile.in
@@ -74,9 +74,13 @@ loaddefs =
 AUTOGENEL = ${loaddefs} ${srcdir}/cus-load.el ${srcdir}/finder-inf.el \
   ${srcdir}/subdirs.el ${srcdir}/eshell/esh-groups.el
 
+ifeq ($(PREFER_ELC),yes)
+BYTE_COMPILE_FLAGS = $(BYTE_COMPILE_EXTRA_FLAGS)
+else
 # Set load-prefer-newer for the benefit of the non-bootstrappers.
 BYTE_COMPILE_FLAGS = \
   --eval '(setq load-prefer-newer t)' $(BYTE_COMPILE_EXTRA_FLAGS)
+endif
 
 # Files to compile before others during a bootstrap.  This is done to
 # speed up the bootstrap process.  They're ordered by size, so we use
@@ -303,9 +307,16 @@ .SUFFIXES:
 # An old-fashioned suffix rule, which, according to the GNU Make manual,
 # cannot have prerequisites.
 ifeq ($(HAVE_NATIVE_COMP),yes)
+ifeq ($(ANCIENT),yes)
+.el.elc:
+	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
+	-l comp -f batch-byte-compile $<
+	touch -t 197001010000 $@
+else
 .el.elc:
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
 	-l comp -f batch-byte+native-compile $<
+endif
 else
 .el.elc:
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $<
diff --git a/src/Makefile.in b/src/Makefile.in
index 04fabd5f42..4b28c01f03 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -914,7 +914,11 @@ $(bootstrap_pdmp):
 	$(RUN_TEMACS) --batch $(BUILD_DETAILS) -l loadup --temacs=pbootstrap \
 		--bin-dest $(BIN_DESTDIR) --eln-dest $(ELN_DESTDIR)
 	@: Compile some files earlier to speed up further compilation.
-	$(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"
+	@: First, byte compile these files, ....
+	ANCIENT=yes $(MAKE) -C ../lisp compile-first \
+	EMACS="$(bootstrap_exe)"
+	@: ... then use them in native compiling these and other files.
+	PREFER_ELC=yes $(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"
 endif
 
 ### Flymake support (for C only)
diff --git a/src/verbose.mk.in b/src/verbose.mk.in
index e3f5678303..01076df946 100644
--- a/src/verbose.mk.in
+++ b/src/verbose.mk.in
@@ -40,12 +40,17 @@ AM_V_CXX     = @$(info $   CXX      $@)
 AM_V_CCLD    = @$(info $   CCLD     $@)
 AM_V_CXXLD   = @$(info $   CXXLD    $@)
 ifeq ($(HAVE_NATIVE_COMP),yes)
-ifeq ($(NATIVE_DISABLED),1)
+ifneq ($(NATIVE_DISABLED),1)
+ifneq ($(ANCIENT),yes)
+AM_V_ELC     = @$(info $   ELC+ELN  $@)
+AM_V_ELN     = @$(info $   ELN      $@)
+else
 AM_V_ELC     = @$(info $   ELC      $@)
 AM_V_ELN =
+endif
 else
-AM_V_ELC     = @$(info $   ELC+ELN  $@)
-AM_V_ELN     = @$(info $   ELN      $@)
+AM_V_ELC     = @$(info $   ELC      $@)
+AM_V_ELN =
 endif
 else
 AM_V_ELC     = @$(info $   ELC      $@)


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Wed, 19 Jan 2022 21:32:30 +0000, Alan Mackenzie <acm@muc.de> said:

    Alan> This was surprisingly difficult to solve.  There appears to be no way in
    Alan> make to set a variable depending on what the target is.  The make manual
    Alan> doesn't say this explicitly, it just depends on vagueness.  After an hour
    Alan> of searching for such a feature, it gradually dawns on you that there is
    Alan> no such feature, even though one might be expected.  I'm glad the Emacs
    Alan> manuals aren't like that.

"Target-specific Variable Values" in the Gnu Make info manual. eg

Makefile:

FOO=bar

default:
	@echo FOO=$(FOO)

foo: FOO=foo
foo:
	@echo FOO=$(FOO)

results:

> make
FOO=bar
> make foo
FOO=foo

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Robert.

On Thu, Jan 20, 2022 at 10:25:26 +0100, Robert Pluim wrote:
> >>>>> On Wed, 19 Jan 2022 21:32:30 +0000, Alan Mackenzie <acm@muc.de> said:

>     Alan> This was surprisingly difficult to solve.  There appears to be no way in
>     Alan> make to set a variable depending on what the target is.  The make manual
>     Alan> doesn't say this explicitly, it just depends on vagueness.  After an hour
>     Alan> of searching for such a feature, it gradually dawns on you that there is
>     Alan> no such feature, even though one might be expected.  I'm glad the Emacs
>     Alan> manuals aren't like that.

> "Target-specific Variable Values" in the Gnu Make info manual. eg

> Makefile:

> FOO=bar

> default:
> 	@echo FOO=$(FOO)

> foo: FOO=foo
> foo:
> 	@echo FOO=$(FOO)

> results:

> > make
> FOO=bar
> > make foo
> FOO=foo

Thank you indeed.  That's exactly what I needed.  I've amended my patch to use
this.

So, if there are no objections, I'll be committing the following to
master within the next day or so:


In early bootstrap, use byte-compiled compiler to native compile first files

This speeds up a make bootstrap by around 15%.

* lisp/Makefile.in (BYTE_COMPILE_FLAGS): set a value specific to compile-first
which doesn't contain the setting of Emacs variable load-prefer-newer.
Add a new make hunk which byte-compiles (rather then native compiles) when the
environment variable ANCIENT is "yes".  Set the date of the .elc files built
to 1970-01-01 to cause a second compilation of them later.

* src/Makefile.in: Add an extra invocation of directory lisp's MAKE with
target compile-first and the flag environment variable ANCIENT set to yes.

* src/verbose.mk.in: When ANCIENT is yes, output ELC, not ELC+ELN for
AM_V_ELC.



diff --git a/lisp/Makefile.in b/lisp/Makefile.in
index 3a72034463..3d03b16331 100644
--- a/lisp/Makefile.in
+++ b/lisp/Makefile.in
@@ -77,6 +77,8 @@ AUTOGENEL =
 # Set load-prefer-newer for the benefit of the non-bootstrappers.
 BYTE_COMPILE_FLAGS = \
   --eval '(setq load-prefer-newer t)' $(BYTE_COMPILE_EXTRA_FLAGS)
+# ... but we must prefer .elc files for those in the early bootstrap.
+compile-first: BYTE_COMPILE_FLAGS = $(BYTE_COMPILE_EXTRA_FLAGS)
 
 # Files to compile before others during a bootstrap.  This is done to
 # speed up the bootstrap process.  They're ordered by size, so we use
@@ -303,9 +305,22 @@ .SUFFIXES:
 # An old-fashioned suffix rule, which, according to the GNU Make manual,
 # cannot have prerequisites.
 ifeq ($(HAVE_NATIVE_COMP),yes)
+ifeq ($(ANCIENT),yes)
+# The first compilation of compile-first, using an interpreted compiler:
+# The resulting .elc files get given a date of 1970-01-01 so that their
+# date stamp is earlier than the source files, causing these to be compiled
+# into native code at the second recursive invocation of this $(MAKE),
+# using these .elc's.  This is faster than just compiling the native code
+# directly using the interpreted compile-first files.
+.el.elc:
+	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
+	-l comp -f batch-byte-compile $<
+	touch -t 197001010000 $@
+else
 .el.elc:
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) \
 	-l comp -f batch-byte+native-compile $<
+endif
 else
 .el.elc:
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $<
diff --git a/src/Makefile.in b/src/Makefile.in
index 04fabd5f42..13392bfad6 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -914,6 +914,9 @@ $(bootstrap_pdmp):
 	$(RUN_TEMACS) --batch $(BUILD_DETAILS) -l loadup --temacs=pbootstrap \
 		--bin-dest $(BIN_DESTDIR) --eln-dest $(ELN_DESTDIR)
 	@: Compile some files earlier to speed up further compilation.
+	@: First, byte compile these files, ....
+	ANCIENT=yes $(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"
+	@: .... then use their .elcs in native compiling these and other files.
 	$(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"
 endif
 
diff --git a/src/verbose.mk.in b/src/verbose.mk.in
index e3f5678303..01076df946 100644
--- a/src/verbose.mk.in
+++ b/src/verbose.mk.in
@@ -40,12 +40,17 @@ AM_V_CXX     = @$(info $   CXX      $@)
 AM_V_CCLD    = @$(info $   CCLD     $@)
 AM_V_CXXLD   = @$(info $   CXXLD    $@)
 ifeq ($(HAVE_NATIVE_COMP),yes)
-ifeq ($(NATIVE_DISABLED),1)
+ifneq ($(NATIVE_DISABLED),1)
+ifneq ($(ANCIENT),yes)
+AM_V_ELC     = @$(info $   ELC+ELN  $@)
+AM_V_ELN     = @$(info $   ELN      $@)
+else
 AM_V_ELC     = @$(info $   ELC      $@)
 AM_V_ELN =
+endif
 else
-AM_V_ELC     = @$(info $   ELC+ELN  $@)
-AM_V_ELN     = @$(info $   ELN      $@)
+AM_V_ELC     = @$(info $   ELC      $@)
+AM_V_ELN =
 endif
 else
 AM_V_ELC     = @$(info $   ELC      $@)


> Robert
> -- 

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Thu, 20 Jan 2022 11:35:17 +0000, Alan Mackenzie <acm@muc.de> said:

    Alan> Thank you indeed.  That's exactly what I needed.  I've amended my patch to use
    Alan> this.

    Alan> So, if there are no objections, I'll be committing the following to
    Alan> master within the next day or so:

I think it could be improved further to avoid having two sub-make
invocations (and remove the need for ANCIENT), but that would be a
bigger and finicky change, so I say go ahead with this for now.

Robert

-- 

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Thu, 20 Jan 2022 14:18:15 +0100, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Thu, 20 Jan 2022 11:35:17 +0000, Alan Mackenzie <acm@muc.de> said:
    Alan> Thank you indeed.  That's exactly what I needed.  I've amended my patch to use
    Alan> this.

    Alan> So, if there are no objections, I'll be committing the following to
    Alan> master within the next day or so:

    Robert> I think it could be improved further to avoid having two sub-make
    Robert> invocations (and remove the need for ANCIENT), but that would be a
    Robert> bigger and finicky change, so I say go ahead with this for now.

One small issue. I have

make --version
GNU Make 4.3
Built for x86_64-pc-linux-gnu

which gives me:

touch -t 197001010000 emacs-lisp/macroexp.elc
make[2]: emacs-lisp/macroexp.elc: Timestamp out of range; substituting 2514-05-30 03:53:03.999999999

Using '197101010000' instead causes no diagnostics.

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Alan Mackenzie]

Hello, Robert.

On Thu, Jan 20, 2022 at 15:54:29 +0100, Robert Pluim wrote:
> >>>>> On Thu, 20 Jan 2022 14:18:15 +0100, Robert Pluim <rpluim@gmail.com> said:

> >>>>> On Thu, 20 Jan 2022 11:35:17 +0000, Alan Mackenzie <acm@muc.de> said:
>     Alan> Thank you indeed.  That's exactly what I needed.  I've amended my patch to use
>     Alan> this.

>     Alan> So, if there are no objections, I'll be committing the following to
>     Alan> master within the next day or so:

>     Robert> I think it could be improved further to avoid having two sub-make
>     Robert> invocations (and remove the need for ANCIENT), but that would be a
>     Robert> bigger and finicky change, so I say go ahead with this for now.

> One small issue. I have

> make --version
> GNU Make 4.3
> Built for x86_64-pc-linux-gnu

> which gives me:

> touch -t 197001010000 emacs-lisp/macroexp.elc
> make[2]: emacs-lisp/macroexp.elc: Timestamp out of range; substituting 2514-05-30 03:53:03.999999999

That's very strange.  It looks like a funny overflow, or something.
Never mind, it's not too important.  But thanks for the heads up.

> Using '197101010000' instead causes no diagnostics.

197101010000 it is then!

I've just committed it.

> Robert
> -- 

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Speeding up the bootstrap build - a quick hack.

[Stefan Monnier]

>> touch -t 197001010000 emacs-lisp/macroexp.elc
>> make[2]: emacs-lisp/macroexp.elc: Timestamp out of range; substituting
>> 2514-05-30 03:53:03.999999999
>
> That's very strange.  It looks like a funny overflow, or something.
> Never mind, it's not too important.  But thanks for the heads up.

Could it just be one of those timezone issues (the timestamp is provided
in local time, and depending on the timezone, I guess this could
correspond to a UTC time in 1969).


        Stefan

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Thu, 20 Jan 2022 17:29:13 -0500, Stefan Monnier <monnier@iro.umontreal.ca> said:

    >>> touch -t 197001010000 emacs-lisp/macroexp.elc
    >>> make[2]: emacs-lisp/macroexp.elc: Timestamp out of range; substituting
    >>> 2514-05-30 03:53:03.999999999
    >> 
    >> That's very strange.  It looks like a funny overflow, or something.
    >> Never mind, it's not too important.  But thanks for the heads up.

    Stefan> Could it just be one of those timezone issues (the timestamp is provided
    Stefan> in local time, and depending on the timezone, I guess this could
    Stefan> correspond to a UTC time in 1969).

Yes, if I stick TZ=UTC in front of the 'touch' there are no
complaints.

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Stephen Leake]

Alan Mackenzie <acm@muc.de> writes:

> This was surprisingly difficult to solve.  There appears to be no way in
> make to set a variable depending on what the target is.

target :: VARIABLE := "value"
targ
Sets VARIABLE to value only while executing target.

I could not find the section in the make manual that says this, but it
must be there somewhere - that's how I learned it :(.

--
-- Stephe

Re: Speeding up the bootstrap build - a quick hack.

[David Engster]

> Alan Mackenzie <acm@muc.de> writes:
>
>> This was surprisingly difficult to solve.  There appears to be no way in
>> make to set a variable depending on what the target is.
>
> target :: VARIABLE := "value"
> targ
> Sets VARIABLE to value only while executing target.
>
> I could not find the section in the make manual that says this, but it
> must be there somewhere - that's how I learned it :(.

It's called target-specific variables:

https://www.gnu.org/software/make/manual/html_node/Target_002dspecific.html

-David

Re: Speeding up the bootstrap build - a quick hack.

[Robert Pluim]

>>>>> On Fri, 21 Jan 2022 02:18:32 -0800, Stephen Leake <stephen_leake@stephe-leake.org> said:

    Stephen> Alan Mackenzie <acm@muc.de> writes:
    >> This was surprisingly difficult to solve.  There appears to be no way in
    >> make to set a variable depending on what the target is.

    Stephen> target :: VARIABLE := "value"
    Stephen> targ
    Stephen> Sets VARIABLE to value only while executing target.

    Stephen> I could not find the section in the make manual that says this, but it
    Stephen> must be there somewhere - that's how I learned it :(.

(info "(make) Target-specific")

Or from the top-level of the Make info file: 'Using variables' ->
'Target specific'.

Robert
-- 

Re: Speeding up the bootstrap build - a quick hack.

[Andrea Corallo]

Hi Alan & all,

while building last master I'm getting several messages like:

Source file ‘.../emacs2/lisp/emacs-lisp/comp.el’ newer than byte-compiled file; using older file
Source file ‘.../emacs2/lisp/emacs-lisp/comp-cstr.el’ newer than byte-compiled file; using older file                                                      

I guess is because of the bootstrap procedure discussed in this thread,
is this expected?

Here my build invocation:

git clean -xfd && ./autogen.sh && ./configure --without-x --with-native-compilation  && time make bootstrap -j16

Best Regards

  Andrea

Re: Speeding up the bootstrap build - a quick hack.

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Date: Mon, 24 Jan 2022 19:43:18 +0000
> Cc: emacs-devel@gnu.org
> 
> Hi Alan & all,
> 
> while building last master I'm getting several messages like:
> 
> Source file ‘.../emacs2/lisp/emacs-lisp/comp.el’ newer than byte-compiled file; using older file
> Source file ‘.../emacs2/lisp/emacs-lisp/comp-cstr.el’ newer than byte-compiled file; using older file                                                      
> 
> I guess is because of the bootstrap procedure discussed in this thread,
> is this expected?

Yes, expected.  We trigger this deliberately, so that .elc files are
used instead of the .el files (to speed up compilation).

Re: Speeding up the bootstrap build - a quick hack.

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Date: Mon, 24 Jan 2022 19:43:18 +0000
>> Cc: emacs-devel@gnu.org
>> 
>> Hi Alan & all,
>> 
>> while building last master I'm getting several messages like:
>> 
>> Source file ‘.../emacs2/lisp/emacs-lisp/comp.el’ newer than byte-compiled file; using older file
>> Source file ‘.../emacs2/lisp/emacs-lisp/comp-cstr.el’ newer than byte-compiled file; using older file                                                      
>> 
>> I guess is because of the bootstrap procedure discussed in this thread,
>> is this expected?
>
> Yes, expected.  We trigger this deliberately, so that .elc files are
> used instead of the .el files (to speed up compilation).

Thanks.  I'm wondering why Emacs complains only about comp.el and
comp-cstr.el given I see we touch a bounch of other files.

  Andrea