Bootstrap optimization

Bootstrap optimization

[Mikael Djurfeldt]

[-- Attachment #1.1: Type: text/plain, Size: 817 bytes --]

I find that the attached patch saves a few minutes for parallel builds on a
four core machine. What it does is to build both eval.go and psyntax.pp.go
serially before the rest is built in parallel (when make is given -j).

Here's an attempt at explanation why this saves time:

Let's denote the .go objects by numbers 1, 2, ... where 1 is eval.go and 2
is psyntax.pp.go. We also prepend a prefix "S" for a slow build with
interpreted psyntax.pp and "F" for a fast build.

Assuming a four core machine, we previously had something like:

S1
S2 S3 S4 S5
     S6 S7 S8 (since S2 builds so slowly)
F9 F10 ...

Now, instead, we have:

S1
S2
F3 F4 F5 F6
...

On my machine, the patch saved 10 minutes out of 55 minutes without the
patch.

To which branch should this be applied, stable-2.2 or master?

Best regards,
Mikael

[-- Attachment #1.2: Type: text/html, Size: 1191 bytes --]

[-- Attachment #2: 0001-Bootstrap-optimization.patch --]
[-- Type: text/x-patch, Size: 1122 bytes --]

From 0fe521100487411a2c48f6af14796a22964844a1 Mon Sep 17 00:00:00 2001
From: Mikael Djurfeldt <mikael@djurfeldt.com>
Date: Thu, 25 Oct 2018 13:53:47 +0200
Subject: [PATCH] Bootstrap optimization

* bootstrap/Makefile.am: Build both eval.go and psyntax-pp.go before the rest
  of the .go files so that they are handled by a fast macro expander. This
  saves time for a parallel build.
---
 bootstrap/Makefile.am | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/bootstrap/Makefile.am b/bootstrap/Makefile.am
index 57b62eb56..3e92ef1af 100644
--- a/bootstrap/Makefile.am
+++ b/bootstrap/Makefile.am
@@ -32,5 +32,8 @@ GUILE_OPTIMIZATIONS = -O1 -Oresolve-primitives
 include $(top_srcdir)/am/bootstrap.am
 
 # We must build the evaluator first, so that we can be sure to control
-# the stack.
-$(filter-out ice-9/eval.go, $(GOBJECTS)): ice-9/eval.go
+# the stack. Then, we build the syntax-case macro expander before the
+# rest, in order to speed up parallel builds.
+ice-9/psyntax-pp.go: ice-9/eval.go
+
+$(filter-out ice-9/eval.go ice-9/psyntax-pp.go, $(GOBJECTS)): ice-9/psyntax-pp.go
-- 
2.11.0

Re: Bootstrap optimization

[Mark H Weaver]

Hi Mikael,

Mikael Djurfeldt <mikael@djurfeldt.com> writes:

> I find that the attached patch saves a few minutes for parallel builds
> on a four core machine. What it does is to build both eval.go and
> psyntax.pp.go serially before the rest is built in parallel (when make
> is given -j).
>
> Here's an attempt at explanation why this saves time:
>
> Let's denote the .go objects by numbers 1, 2, ... where 1 is eval.go
> and 2 is psyntax.pp.go. We also prepend a prefix "S" for a slow build
> with interpreted psyntax.pp and "F" for a fast build.
>
> Assuming a four core machine, we previously had something like:
>
> S1
> S2 S3 S4 S5
>      S6 S7 S8 (since S2 builds so slowly)
> F9 F10 ...
>
> Now, instead, we have:
>
> S1
> S2
> F3 F4 F5 F6
> ...
>
> On my machine, the patch saved 10 minutes out of 55 minutes without
> the patch.

Yes, I've also noticed this issue, and long thought that it would be
better to serialize building several of the early bootstrap .go objects.

However ...

> From 0fe521100487411a2c48f6af14796a22964844a1 Mon Sep 17 00:00:00 2001
> From: Mikael Djurfeldt <mikael@djurfeldt.com>
> Date: Thu, 25 Oct 2018 13:53:47 +0200
> Subject: [PATCH] Bootstrap optimization
>
> * bootstrap/Makefile.am: Build both eval.go and psyntax-pp.go before the rest
>   of the .go files so that they are handled by a fast macro expander. This
>   saves time for a parallel build.
> ---
>  bootstrap/Makefile.am | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/bootstrap/Makefile.am b/bootstrap/Makefile.am
> index 57b62eb56..3e92ef1af 100644
> --- a/bootstrap/Makefile.am
> +++ b/bootstrap/Makefile.am
> @@ -32,5 +32,8 @@ GUILE_OPTIMIZATIONS = -O1 -Oresolve-primitives
>  include $(top_srcdir)/am/bootstrap.am
>  
>  # We must build the evaluator first, so that we can be sure to control
> -# the stack.
> -$(filter-out ice-9/eval.go, $(GOBJECTS)): ice-9/eval.go
> +# the stack. Then, we build the syntax-case macro expander before the
> +# rest, in order to speed up parallel builds.
> +ice-9/psyntax-pp.go: ice-9/eval.go
> +
> +$(filter-out ice-9/eval.go ice-9/psyntax-pp.go, $(GOBJECTS)): ice-9/psyntax-pp.go

The downside of this approach to serialization is that when we add file
X.scm to the list of objects to build serially, we force a full rebuild
whenever X.scm is modified.  At present, eval.scm is the only file that
forces a full rebuild.  Your patch would add psyntax-pp.scm to that
list.

I don't feel strongly about it, and maybe your patch is still a net
benefit, but I very much wish we had a better way to optimize the early
bootstrap without adding these bogus dependencies.

> To which branch should this be applied, stable-2.2 or master?

If we decide to adopt this approach, it should probably be committed to
stable-2.2, but first I'd like to give other people an opportunity to
share their thoughts on this.

Thoughts?

      Mark

Re: Bootstrap optimization

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Den sön 28 okt. 2018 02:35Mark H Weaver <mhw@netris.org> skrev:

> The downside of this approach to serialization is that when we add file
> X.scm to the list of objects to build serially, we force a full rebuild
> whenever X.scm is modified.  At present, eval.scm is the only file that
> forces a full rebuild.  Your patch would add psyntax-pp.scm to that
> list.
>
> I don't feel strongly about it, and maybe your patch is still a net
> benefit, but I very much wish we had a better way to optimize the early
> bootstrap without adding these bogus dependencies.
>
> > To which branch should this be applied, stable-2.2 or master?
>
> If we decide to adopt this approach, it should probably be committed to
> stable-2.2, but first I'd like to give other people an opportunity to
> share their thoughts on this.
>
> Thoughts?


Thank you for spotting this, Mark. I didn't think of this.

There seems to be a simple solution in using order-only prerequisites. I'll
test this later and suggest a new patch.

Best regards,
Mikael

[-- Attachment #2: Type: text/html, Size: 1534 bytes --]

Re: Bootstrap optimization

[Mikael Djurfeldt]

[-- Attachment #1.1: Type: text/plain, Size: 374 bytes --]

OK, here's a new patch. OK to apply it?

This actually also fixes the existing problem of all bootstrap objects
being rebuilt of eval.scm is touched.

The patch is verified to give a faster build for 4 and 32 build threads.

The only downside is that it requires GNU Make 3.8 (which was released
2002) or later, but that shouldn't be a problem, right?

Best regards,
Mikael

[-- Attachment #1.2: Type: text/html, Size: 525 bytes --]

[-- Attachment #2: 0001-Bootstrap-optimization.patch --]
[-- Type: text/x-patch, Size: 1131 bytes --]

From 332471874aa227e8b25b747f560ab9185af4f2fb Mon Sep 17 00:00:00 2001
From: Mikael Djurfeldt <mikael@djurfeldt.com>
Date: Thu, 25 Oct 2018 13:53:47 +0200
Subject: [PATCH] Bootstrap optimization

* bootstrap/Makefile.am: Build both eval.go and psyntax-pp.go before
  the rest of the .go files so that they are handled by a fast macro
  expander. This saves time for a parallel build.
---
 bootstrap/Makefile.am | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/bootstrap/Makefile.am b/bootstrap/Makefile.am
index 57b62eb56..bcb22cdbc 100644
--- a/bootstrap/Makefile.am
+++ b/bootstrap/Makefile.am
@@ -32,5 +32,9 @@ GUILE_OPTIMIZATIONS = -O1 -Oresolve-primitives
 include $(top_srcdir)/am/bootstrap.am
 
 # We must build the evaluator first, so that we can be sure to control
-# the stack.
-$(filter-out ice-9/eval.go, $(GOBJECTS)): ice-9/eval.go
+# the stack. Then, we build the syntax-case macro expander before the
+# rest, in order to speed up parallel builds.
+ice-9/psyntax-pp.go: | ice-9/eval.go
+
+$(filter-out ice-9/eval.go ice-9/psyntax-pp.go, $(GOBJECTS)): | \
+	ice-9/psyntax-pp.go
-- 
2.11.0

Officially require GNU Make to build Guile? (was Re: Bootstrap optimization)

[Mark H Weaver]

Hi Mikael,

Mikael Djurfeldt <mikael@djurfeldt.com> writes:

> OK, here's a new patch. OK to apply it?
>
> This actually also fixes the existing problem of all bootstrap objects
> being rebuilt of eval.scm is touched.

Very nice!  I didn't know about this feature in GNU Make.

> The patch is verified to give a faster build for 4 and 32 build threads.
>
> The only downside is that it requires GNU Make 3.8 (which was released
> 2002) or later, but that shouldn't be a problem, right?

Hmm, good question.  I'm not sure, not because 2002 is too recent, but
rather because I'm not sure that GNU Make should be a requirement for
building Guile.

Guile's README does not list GNU Make in the section of "Required
External Packages".  This, along with the fact that we use Automake
which is clearly designed to produce portable Makefiles, makes me
inclined to think that if Guile depends on non-standard extensions in
GNU Make, that this is a bug.

On the other hand, if this is a bug, it seems that we've had this bug
for several years at least, and that non-GNU systems are already working
around it by adding GNU Make as a requirement.  For example, the Guile 2
packages in the OpenBSD and FreeBSD ports collections already list GNU
Make as a prerequisite for building Guile 2.  Our README also has
"Special Instructions For Some Systems" which mentions that gmake is
required on FreeBSD 11.0.  However, there's no mention of any other
non-GNU systems requiring GNU Make to build Guile.

So, we now have a choice.  We can fully embrace a requirement on GNU
Make, or we can treat it as a bug to be fixed.

I'm still inclined to consider it a bug, but maybe we can have the best
of both worlds here.  I see that Automake has conditionals:

  https://www.gnu.org/software/automake/manual/automake.html#Conditionals

How hard would it be to test for GNU Make in our configure script, and
then to use your improved Makefile rules only when GNU Make is present?

    Regards,
      Mark

Re: Officially require GNU Make to build Guile? (was Re: Bootstrap optimization)

[Greg Troxel]

Mark H Weaver <mhw@netris.org> writes:

> Hmm, good question.  I'm not sure, not because 2002 is too recent, but
> rather because I'm not sure that GNU Make should be a requirement for
> building Guile.
>
> Guile's README does not list GNU Make in the section of "Required
> External Packages".  This, along with the fact that we use Automake
> which is clearly designed to produce portable Makefiles, makes me
> inclined to think that if Guile depends on non-standard extensions in
> GNU Make, that this is a bug.

Agreed that requiring GNU Make is a bug.

> On the other hand, if this is a bug, it seems that we've had this bug
> for several years at least, and that non-GNU systems are already working
> around it by adding GNU Make as a requirement.  For example, the Guile 2
> packages in the OpenBSD and FreeBSD ports collections already list GNU
> Make as a prerequisite for building Guile 2.  Our README also has
> "Special Instructions For Some Systems" which mentions that gmake is
> required on FreeBSD 11.0.  However, there's no mention of any other
> non-GNU systems requiring GNU Make to build Guile.

NetBSD's pkgsrc (which is portable to about 20 systems) also is marked
to need GNU make.  Practically, it's not a problem, because packaging
systems have to deal with many programs that require GNU make anway.

> So, we now have a choice.  We can fully embrace a requirement on GNU
> Make, or we can treat it as a bug to be fixed.
>
> I'm still inclined to consider it a bug, but maybe we can have the best
> of both worlds here.  I see that Automake has conditionals:

I am also inclined to see it as a bug.  Generally, most times GNU make
is needed are because someone used an extension without realizing that
there is make other than GNU make, vs it making a huge difference.  But
if it is hard to do things differently, that's somewhat tilting at
windwills.   But hey, that's what running scheme is all about!

Re: Officially require GNU Make to build Guile? (was Re: Bootstrap optimization)

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

On Sun, Oct 28, 2018 at 11:34 PM Mark H Weaver <mhw@netris.org> wrote:

> I'm still inclined to consider it a bug, but maybe we can have the best
> of both worlds here.  I see that Automake has conditionals:
>
>   https://www.gnu.org/software/automake/manual/automake.html#Conditionals
>
> How hard would it be to test for GNU Make in our configure script, and
> then to use your improved Makefile rules only when GNU Make is present?
>

I don't think that it is hard in itself. However, it is hard for me in this
case since I don't know how you have been thinking with regard to the
structure of the build system. E.g., I don't know why there's an am/
bootstrap.am which is included in bootstrap/Makefile.am rather than having
that material in Makefile.am. In addition, I think it is better that I
spend the little time I have in other ways---sorry.

But, as you concluded, Guile currently uses GNU Make specific
functionality. $(filter-out ...) in bootstrap/Makefile.am is such a case
and also the vpath and %-thingies in am/bootstrap.am. Probably, you should
start out by making an inventory of the various uses of GNU Make
functionality and for each case evaluate how much work would be required to
make it standard. First then it is possible to decide if it is worth to do
it.

However, note that applying my suggested patch would not really change the
situation much in this respect since that piece of functionality already
now depends on the GNU Make specific $(filter-out ...), such that when GNU
Make specifics get handled, having applied my patch doesn't really increase
the amount of work in any way.

Best regards,
Mikael

[-- Attachment #2: Type: text/html, Size: 2195 bytes --]

Re: Officially require GNU Make to build Guile? (was Re: Bootstrap optimization)

[Mark H Weaver]

Hi Mikael,

Mikael Djurfeldt <mikael@djurfeldt.com> writes:

> But, as you concluded, Guile currently uses GNU Make specific
> functionality. $(filter-out ...) in bootstrap/Makefile.am is such a
> case and also the vpath and %-thingies in am/bootstrap.am.

I guess you're right.

> Probably, you should start out by making an inventory of the various
> uses of GNU Make functionality and for each case evaluate how much
> work would be required to make it standard. First then it is possible
> to decide if it is worth to do it.
>
> However, note that applying my suggested patch would not really change
> the situation much in this respect since that piece of functionality
> already now depends on the GNU Make specific $ (filter-out ...), such
> that when GNU Make specifics get handled, having applied my patch
> doesn't really increase the amount of work in any way.

It's a good point.  Okay, I'm convinced.  Please go ahead and push your
patch to stable-2.2.  Thanks for talking me through it.

     Regards,
       Mark