Re: Bootstrap optimization

[Mark H Weaver <mhw@netris.org>]

Hi Mikael,

Mikael Djurfeldt <mikael@djurfeldt.com> writes:

> I find that the attached patch saves a few minutes for parallel builds
> on a four core machine. What it does is to build both eval.go and
> psyntax.pp.go serially before the rest is built in parallel (when make
> is given -j).
>
> Here's an attempt at explanation why this saves time:
>
> Let's denote the .go objects by numbers 1, 2, ... where 1 is eval.go
> and 2 is psyntax.pp.go. We also prepend a prefix "S" for a slow build
> with interpreted psyntax.pp and "F" for a fast build.
>
> Assuming a four core machine, we previously had something like:
>
> S1
> S2 S3 S4 S5
>      S6 S7 S8 (since S2 builds so slowly)
> F9 F10 ...
>
> Now, instead, we have:
>
> S1
> S2
> F3 F4 F5 F6
> ...
>
> On my machine, the patch saved 10 minutes out of 55 minutes without
> the patch.

Yes, I've also noticed this issue, and long thought that it would be
better to serialize building several of the early bootstrap .go objects.

However ...

> From 0fe521100487411a2c48f6af14796a22964844a1 Mon Sep 17 00:00:00 2001
> From: Mikael Djurfeldt <mikael@djurfeldt.com>
> Date: Thu, 25 Oct 2018 13:53:47 +0200
> Subject: [PATCH] Bootstrap optimization
>
> * bootstrap/Makefile.am: Build both eval.go and psyntax-pp.go before the rest
>   of the .go files so that they are handled by a fast macro expander. This
>   saves time for a parallel build.
> ---
>  bootstrap/Makefile.am | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/bootstrap/Makefile.am b/bootstrap/Makefile.am
> index 57b62eb56..3e92ef1af 100644
> --- a/bootstrap/Makefile.am
> +++ b/bootstrap/Makefile.am
> @@ -32,5 +32,8 @@ GUILE_OPTIMIZATIONS = -O1 -Oresolve-primitives
>  include $(top_srcdir)/am/bootstrap.am
>  
>  # We must build the evaluator first, so that we can be sure to control
> -# the stack.
> -$(filter-out ice-9/eval.go, $(GOBJECTS)): ice-9/eval.go
> +# the stack. Then, we build the syntax-case macro expander before the
> +# rest, in order to speed up parallel builds.
> +ice-9/psyntax-pp.go: ice-9/eval.go
> +
> +$(filter-out ice-9/eval.go ice-9/psyntax-pp.go, $(GOBJECTS)): ice-9/psyntax-pp.go

The downside of this approach to serialization is that when we add file
X.scm to the list of objects to build serially, we force a full rebuild
whenever X.scm is modified.  At present, eval.scm is the only file that
forces a full rebuild.  Your patch would add psyntax-pp.scm to that
list.

I don't feel strongly about it, and maybe your patch is still a net
benefit, but I very much wish we had a better way to optimize the early
bootstrap without adding these bogus dependencies.

> To which branch should this be applied, stable-2.2 or master?

If we decide to adopt this approach, it should probably be committed to
stable-2.2, but first I'd like to give other people an opportunity to
share their thoughts on this.

Thoughts?

      Mark