[PATCH] Speed up stage0 bootstrap build using prebuilts

[PATCH] Speed up stage0 bootstrap build using prebuilts

[Jonas Hahnfeld via Developers list for Guile, the GNU extensibility library]

[-- Attachment #1: Type: text/plain, Size: 1366 bytes --]

From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17 00:00:00 2001
From: Jonas Hahnfeld <hahnjo@hahnjo.de>
Date: Thu, 4 Jan 2024 11:44:55 +0100
Subject: [PATCH] Speed up stage0 bootstrap build using prebuilts

Use prebuilt bytecode of ice-9/eval.go and others for all of stage0,
it is optimized and evaluation is much faster. In my environment,
this speeds up the build of guile-3.0.9 from around 29 minutes to
only 19 minutes.

* meta/build-env.in: In stage0, prefer prebuilt bytecode over just
compiled stage0 files.
---
 meta/build-env.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/build-env.in b/meta/build-env.in
index bdc88ded4..446a536af 100644
--- a/meta/build-env.in
+++ b/meta/build-env.in
@@ -58,7 +58,7 @@ then
     fi
     export GUILE_LOAD_PATH
     case "$GUILE_BOOTSTRAP_STAGE" in
-        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
+        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;
         stage1) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage1:${top_builddir}/stage0" ;;
         stage2) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage1" ;;
         *) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage2:${top_builddir}/guile-readline" ;;
-- 
2.43.0

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] Speed up stage0 bootstrap build using prebuilts

[Jonas Hahnfeld via Developers list for Guile, the GNU extensibility library]

[-- Attachment #1: Type: text/plain, Size: 1615 bytes --]

Dear Guile maintainers,

please consider below patch to speed up the build from release tarballs
that include prebuilt bytecode.

Jonas

On Thu, 2024-01-04 at 11:57 +0100, Jonas Hahnfeld wrote:
> From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17 00:00:00 2001
> From: Jonas Hahnfeld <hahnjo@hahnjo.de>
> Date: Thu, 4 Jan 2024 11:44:55 +0100
> Subject: [PATCH] Speed up stage0 bootstrap build using prebuilts
> 
> Use prebuilt bytecode of ice-9/eval.go and others for all of stage0,
> it is optimized and evaluation is much faster. In my environment,
> this speeds up the build of guile-3.0.9 from around 29 minutes to
> only 19 minutes.
> 
> * meta/build-env.in: In stage0, prefer prebuilt bytecode over just
> compiled stage0 files.
> ---
>  meta/build-env.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/build-env.in b/meta/build-env.in
> index bdc88ded4..446a536af 100644
> --- a/meta/build-env.in
> +++ b/meta/build-env.in
> @@ -58,7 +58,7 @@ then
>      fi
>      export GUILE_LOAD_PATH
>      case "$GUILE_BOOTSTRAP_STAGE" in
> -        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
> +        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;
>          stage1) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage1:${top_builddir}/stage0" ;;
>          stage2) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage1" ;;
>          *) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage2:${top_builddir}/guile-readline" ;;


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: bug#69314: [PATCH] Speed up stage0 bootstrap build using prebuilts

[Ludovic Courtès]

Hi Jonas,

Jonas Hahnfeld <hahnjo@hahnjo.de> skribis:

> On Thu, 2024-01-04 at 11:57 +0100, Jonas Hahnfeld wrote:
>> From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17 00:00:00 2001
>> From: Jonas Hahnfeld <hahnjo@hahnjo.de>
>> Date: Thu, 4 Jan 2024 11:44:55 +0100
>> Subject: [PATCH] Speed up stage0 bootstrap build using prebuilts
>> 
>> Use prebuilt bytecode of ice-9/eval.go and others for all of stage0,
>> it is optimized and evaluation is much faster. In my environment,
>> this speeds up the build of guile-3.0.9 from around 29 minutes to
>> only 19 minutes.
>> 
>> * meta/build-env.in: In stage0, prefer prebuilt bytecode over just
>> compiled stage0 files.
>> ---
>>  meta/build-env.in | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/meta/build-env.in b/meta/build-env.in
>> index bdc88ded4..446a536af 100644
>> --- a/meta/build-env.in
>> +++ b/meta/build-env.in
>> @@ -58,7 +58,7 @@ then
>>      fi
>>      export GUILE_LOAD_PATH
>>      case "$GUILE_BOOTSTRAP_STAGE" in
>> -        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
>> +        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;

I don’t understand why changing the order would make a difference.
Surely if .go files are available under prebuilt/, they’ll be found,
even if that directory comes second?  Or am I missing something?

Thanks for the patch!

Ludo’.

Re: bug#69314: [PATCH] Speed up stage0 bootstrap build using prebuilts

[Jonas Hahnfeld via Developers list for Guile, the GNU extensibility library]

[-- Attachment #1: Type: text/plain, Size: 1902 bytes --]

On Mon, 2024-05-06 at 11:34 +0200, Ludovic Courtès wrote:
> Hi Jonas,
> 
> Jonas Hahnfeld <hahnjo@hahnjo.de> skribis:
> 
> > On Thu, 2024-01-04 at 11:57 +0100, Jonas Hahnfeld wrote:
> > > From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17 00:00:00 2001
> > > From: Jonas Hahnfeld <hahnjo@hahnjo.de>
> > > Date: Thu, 4 Jan 2024 11:44:55 +0100
> > > Subject: [PATCH] Speed up stage0 bootstrap build using prebuilts
> > > 
> > > Use prebuilt bytecode of ice-9/eval.go and others for all of stage0,
> > > it is optimized and evaluation is much faster. In my environment,
> > > this speeds up the build of guile-3.0.9 from around 29 minutes to
> > > only 19 minutes.
> > > 
> > > * meta/build-env.in: In stage0, prefer prebuilt bytecode over just
> > > compiled stage0 files.
> > > ---
> > >  meta/build-env.in | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/meta/build-env.in b/meta/build-env.in
> > > index bdc88ded4..446a536af 100644
> > > --- a/meta/build-env.in
> > > +++ b/meta/build-env.in
> > > @@ -58,7 +58,7 @@ then
> > >      fi
> > >      export GUILE_LOAD_PATH
> > >      case "$GUILE_BOOTSTRAP_STAGE" in
> > > -        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
> > > +        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;
> 
> I don’t understand why changing the order would make a difference.
> Surely if .go files are available under prebuilt/, they’ll be found,
> even if that directory comes second?  Or am I missing something?

Yes, they are found but with lower priority. As I try to explain in the
commit message (could be improved though), the same bytecode is also
built during stage0, but without optimizations which makes the rest of
stage0 awfully slow.

Jonas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: bug#69314: [PATCH] Speed up stage0 bootstrap build using prebuilts

[Simon Tournier]

Hi Ludo,

On lun., 06 mai 2024 at 11:34, Ludovic Courtès <ludo@gnu.org> wrote:

>>> -        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
>>> +        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;
>
> I don’t understand why changing the order would make a difference.
> Surely if .go files are available under prebuilt/, they’ll be found,
> even if that directory comes second?  Or am I missing something?

Well, depending on what had been built before, it might read:

--8<---------------cut here---------------start------------->8---
$ ls -1 {stage0,prebuilt/x86_64-unknown-linux-gnu}/ice-9/{boot-9,eval,psyntax-pp}.go
prebuilt/x86_64-unknown-linux-gnu/ice-9/boot-9.go
prebuilt/x86_64-unknown-linux-gnu/ice-9/eval.go
prebuilt/x86_64-unknown-linux-gnu/ice-9/psyntax-pp.go
stage0/ice-9/boot-9.go
stage0/ice-9/eval.go
stage0/ice-9/psyntax-pp.go
--8<---------------cut here---------------end--------------->8---

Therefore, the order in GUILE_LOAD_COMPILED_PATH matters, no?

Somehow, similarly as:

        stage1) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage1:${top_builddir}/stage0" ;;

I would set the expected most optimized first, such that:

        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;

assuming ’prebuilt’ provides something more optimized than ’stage0’.


Cheers,
simon